Attend It Again: Recurrent Attention Convolutional Neural Network for Action Recognition

: Human action recognition in videos is an important task with a broad range of applications. In this study, we improve the performance of recurrent attention convolutional neural network (RACNN) by proposing a novel model, “attention-again”. We consider the nature of video frames as sequences, which will cause the change of regions of interest in the frame, thus we cannot use an attention mechanism similar to that in images. “Attention-again” model is a variant from traditional attention model for recognizing human activities and is embedded in two long short-term memory (LSTM) layers. Different from hierarchal LSTM which change the LSTM structure to combine the hidden states from two LSTM layers, our proposals introduce “attention-again” model to avoid the change of LSTM structure. Furthermore, this model not only learns the relations in each frame, but also obtains the relations among all frames, and these relations instruct the next learning stage. Therefore, our proposed model outperform the baseline and is superior to methods with the same experimental conditions on three benchmark datasets: UCF-11, HMDB-51 and UCF-101. To understand how the model works, we also visualize the region of interest in the


Introduction
Human action recognition in videos is one of the fundamental challenging tasks in computer vision. Since studies in images have had great success, researchers are paying more attention to video. In the literature, recognizing actions plays a vital role in several applications, such as multimedia contexts [1,2], video surveillance [3,4], video streaming [5][6][7], health care systems [8] and smart indoor security systems [9][10][11]. Recently, human action recognition has achieved great success in depth video and sensor [12][13][14][15][16][17][18]. In this work, we focus on RGB video sequences from single camera.
Unlike action recognition in images, action recognition in videos relies on motion dynamics in addition to visual appearance. Traditional approaches use hand-crafted features, and then input the aggregated features to the classifier. In contrast, the recent surge of deep neural networks (DNNs) achieve a significant advancement in extracting features. Among them, convolutional neural networks (CNNs) have shown a great ability to produce a rich representation of the input and achieve promising results on several challenging tasks, including image classification, object detection, machine translation, and tracking [19][20][21]. For this task, a direct approach to recognize action is to arm the convolutional operation with temporal information such as 3D convolutional networks [22,23]. Although they acheive great performance, these works are still missing much temporal information. Recurrent neural networks (RNNs), another branch of DNNs, are a class of artificial neural networks, where the recurrent architecture allows the networks to exhibit dynamic temporal behavior. Since the video frames are sequential, RNNs might be more suitable for this kind of task. Thus, many works [24,25] have combined RNN with CNN for action recognition and have shown good results. Specifically, LSTM [26] has been proven by Donahue et al. [27], Shi et al. [28] and Ng et al. [29] to be effective for action recognition task from video sequences. In [25], they stack three LSTM layers and get great performance, however, they only fed the bottom hidden states of LSTM to top layers. Baradel et al. [30] are aware of weakness of this simply structure, thus, they embed the input feature maps with spatial attention model and change the structure of LSTM to fit the different input. Moreover, these changes improve the performance and then [30] outperform [25]. Attention mechanism mentioned in above method has obtained many interests from computer vision community and received a great achievement in caption generation, neural machine translation and image recognition. By attention mechanism, a model can learn where should be paid attention to, for example, in Figure 2 , the boy is juggling soccer, attention mechanism leads the model to focus on the leg and football which are important factors to classify the video into "soccer_juggling". The attention mechanism due to great performance of catching region of interest is also introduced to action recognition [25,27,31]. For example, Sharma et al. [25] transferred the attention mechanism on the spatial domain to action recognition.
Although the above DNN architectures with attention mechanisms have achieved great performance, we believe that attention mechanisms in video are not the same as that in images. For instance, in the task of image caption, the model with attention mechanisms learns to pay attention to the part of image from the previous state, which focus on the other part of image. However, images are different from videos, which consist of many frames. Therefore, the regions where activities are happening will be changing, and the interesting areas in previous frames might not be the main portions in next video frame. In addition, the intermediate feature vectors are limited as they are generated by only looking at previous frames in order. If we simply utilize this attention mechanisms into videos, we would only focus on the spatial information, and take less global state into consideration. In [32], they inspired by human behavior of reading document that they cannot understand some words in first read. Therefore, they introduce a simple mechanism that first reads the input sequence before committing to a representation of each word, and then, they propose a simple copy mechanism that is able to exploit very small vocabularies. Taking this mechanism into account, we find that understanding videos is also similar to reading, thus, we aim to develop a model to obtain great performances for action recognition in videos.
Therefore, in this study, we propose "attention-again" model to combine the neighboring frames with current frame to get the interesting part of the frame. This model has two LSTM layers. In the bottom layer of LSTM, we use the traditional attention mechanisms and generate the hidden state of LSTM unit from previous hidden state and current input. Next step, we integrate the hidden state of previous LSTM unit in top layer, current input feature and the current output from the bottom layer of LSTM unit. By this way, we could capture spatial relations by bottom LSTM layer and obtain more global information from top LSTM layer. Consequently, we not only consider the information from current state, but also fuse context information. Besides, we incorporate our proposed "attention-again" model with DNN framework. Our architecture is named recurrent attention convolutional Neural Network (RACNN) and is shown in Figure 1, where CNNs encode the input video frames into feature sequences.
The contributions of our proposals can be summarized as follows: (1) We introduce a novel deep learning model "attention-again" for action recognition in videos which can pay attention to the important region in video. Compared with traditional attention mechanisms, this model has improved performance. (2) An extensive set of experiments were conducted to demonstrate that the proposed model "attention-again" outperform the methods with the same experimental condition on three benchmark datasets: UCF-11 [33], HMDB-51 [34] and UCF-101 [35].
The rest of the paper is organized as following: In Section 2, we review the related work. We also present the development of action recognition and show some state-of-the-art work. The proposed model and the principle of our method is presented in Section 3. Experimental results and some implied training details are discussed in Section 4. Finally, Section 5 concludes this paper. Traditional attention model in bottom layer refine the input feature sequences. The proposed model "attention-again" will get more information from input and hidden state.

Related Work
Human action recognition is a longstanding topic in computer vision community and is a well studied problem with various standard benchmarks. The literature on action recognition in video is vast and too broad for us to cover here completely. In this work, we present some related works on ConvNet architecture, LSTM-like architecture and attention mechanism.

ConvNet Architecture
Following the great success of CNNs for image classification, image segmentation and other computer vision tasks, deep learning algorithms have been used in video based human action recognition and have been proven that features learned from CNN are much better than hand-crafted features. In contrast to hand-craft shallow video representation, recent efforts try to learn the representation of video automatically from large scale labeled video data. For example, Karpathy et al. [23] directly applied CNNs to multiple frames in each sequence and obtain the temporal relations by pooling using single, late, early and slow fusion. Nevertheless, the results of this scheme are just marginally better than those of a single frame baseline, and indicate that motion features are difficult to obtain by simply and directly pooling spatial features from CNNs. To ameliorate this shortcoming, Simonyan et al. [36] proposed a two-stream ConvNet architecture, which combines optical flow and RGB video frames to train CNN and achieves comparable results with the state-of-the-art hand-craft based methods. Furthermore, dense trajectory pool CNN that combines improved Dense Trajectories (iDT) [37] and two stream CNNs via the pooling layer achieves the state-of-the-art performance. Wang et al. [38] proposed a novel framework, temporal segment network (TSN), inspired from long-range temporal structure modeling, and obtained the state-of-the-art performance on the datasets of HMDB51 and UCF101. Another paradigm of action recognition, 3DCNNs, when solving above issues receive much attention from the community, which regard the video as a spatiotemporal blobs and train 3D filters to recognizing action. 3DCNNs is first introduced by Ji et al. [22] and learned 3D convolution kernels in both spatial and temporal space based on a straightforward extension of the established 2DCNNs. However, the performance of this method has been harder to scale for multi-stream methods. CNN based methods cannot accurately model the dynamics by simply averaging the scores across the temporal domain, even if the appearance features already achieve remarkable performance on other computer vision tasks.

LSTM-Like Architecture
To model the dynamics between frames, RNNs, particularly LSTM, have been considered in video based human action recognition. LSTMs are recurrent modules which can learn long-term dependencies using a hidden state augmented with nonlinear mechanisms to allow the state to propagate without modification. Most recent activity recognition methods [25,27,29] have CNNs underlying the LSTMs to mine the information in the feature map. Donahue et al. [27] proposed LSTMs that explicitly model short snippets of ConvNet activations. Meanwhile, Ng et al. [29] demonstrated that two-stream LSTMs outperform improved dense trajectories (iDT) [39] and two-stream CNNs, although they need to pre-train their architecture on sports 1-M videos. Srivastava et al. [24] also utilized pre-training on hundreds of hours of sports video, but they proposed an interesting LSTM based unsupervised training method and then fine-tuned this unsupervised pre-training LSTM to adapt human action recognition tasks. More importantly, they used an encoder LSTM to map an input sequence into a fixed-length representation, and used single or multiple decoder LSTMs to perform reconstruction of the input sequence or prediction of the future sequence. Then, Mahasseni et al. [40] proposed training LSTMs that are regularized using the output of another encoder LSTM (RLSTM-g3) grounded on 3D human-skeleton training data. Baccouche et al. [41] proposed a two-step learning method in which 3D features are extracted from CNNs, and then an RNN is trained to classify each sequence by considering the temporal evolution of the learned features.

Attention Mechanism
To address this, attention models was introduced to capture where the model is focusing its attention when performing this task. Karpathy et al. [23] used a multi-resolution CNN architecture to perform action recognition in videos. They mentioned the concept of fovea but they fixed attention to the center of the frame. More recently, Jaderberg et al. [42] proposed a soft-attention mechanism called the Spatial Transformer module which they added between the layers of CNNs. Instead of weighting locations using a softmax layer, which we do, they applied affine transformations to multiple layers of their CNN to attend to the relevant part and get state-of-the-art results on the Street View House Numbers dataset. Yeung et al. [43] performed dense action labeling using a temporal attention based model on the input-output context and report higher accuracy and better understanding of temporal relationships in action videos. Fantastic work by Sharma [25] and Xu et al. [44] used both soft attention and hard attention mechanisms to generate descriptions. Our work directly builds upon these works. Iin this paper, we use a soft attention mechanism for action recognition in videos. Combined with proposed "attention-again" model, our model outperforms theirs.

The Proposed Method
The overall architecture of the proposed method is illustrated in Figure 1. We will describe the three major components of our method in detail in this section: (1) convolutional feature extraction; (2) LSTM sequence modeling and attention model; and (3) "attention-again" model.

Convolutional Features Extractor
CNN has shown a good ability to produce a rich representation of the input image by embedding it into a fixed feature vector. In many tasks, it is natural to use CNN as the encoder to extract the features and networks which are pre-trained on ImageNet dataset might obtain great performance. In general, our architecture can adapt any deep convolution networks for feature extraction, thus, we conduct our encoder network with GoogLeNet, VGG and ResNet-101,152.
First, we employ the opencv toolkit to split each video into 16 video frames with fixed interval, and then, we resize all frames into 224 × 224 and extract feature cubes from the different layers in different networks. For example, in GoogLeNet, we pick up the last convolutional layer, and we extract feature cubes from the last pooling layer and in both ResNet-101 and ResNet-152, also known as , block4 unit3 conv2. Simply, we represent these feature cubes as D × K × K (e.g., 512 × 14 × 14 used in VGG Net). Especially, K × K is the size of feature map of the input image and D is the number of the feature dimensions. Hence, at each time-step t, we extract K 2 D-dimensional vectors. We refer to these vectors as feature slices into a feature cube, and denoted as: where each vertical feature vector is the same region of feature map in different dimensions. At time step t, these K 2 vertical feature slices to different overlapping regions in the input space and our model chooses to focus its attention on these K 2 regions.

LSTM Sequence Modeling
Since the video is a sequence of frames, we establish a model p(y|X t , h t−1 ) with RNNs. After extracting the features in videos, we input all X t in LSTMs. The LSTMs have been proven to get great performance in many tasks, thus, we utilize LSTMs to process the features. In this subsection, we briefly describe the LSTMs unit which is the basic part of our model. LSTM relies on a fantastic structure of gates (i t , f t , o t , g t ) to control the flow of information to the hidden neurons and has the ability to preserve sequence information over time and capture long-term dependencies. Moreover, recent advances in computer vision also suggest that LSTM has potential to model videos for action recognition [25]. We follow the LSTM implementation in [45], which is given as follows: where i t is the input gate, f t is the forget gate, o t is the output gate, and g t is calculated, as shown in Equation (2). c t is the memory cell state at t, h t is the hidden state at t, and x n,t stands for input features at t. σ means the sigmoid function and denotes the element-wise multiplication. The weight matrices are denoted by W ij and biases b j , which are the trainable parameters. The cores of the LSTM unit are a memory cell c t and three gates i t , o t , and f t . Memory cell c t encodes the information of previous memory cell c t−1 and current input by employing these three gates, which control the flow of information in LSTM units. For convenience, we denote h t , c t = LSTM(x t , h t−1 , c t−1 ) as the computation function for updating the LSTM internal state.
In [42], they conduct a hierarchical structure with two LSTM layers, which is similar to our method. However, they change the structure of LSTM to fit different input. Compared with [42], we take the traditional LSTM as our loop-connected processor, and attention-again model is used to solve the problem of input state.
To converge faster, we use the following initialization strategy (see Xu et al. [44]) for the cell state and the hidden state of the LSTM: where f init,c and f init,h are two multilayer perceptrons and T is the number of time-steps in the model. These values are used to calculate the first location softmax a 1 t,i (see Section 3.2.2) which determines the initial input x n,1 .

Attention Model
In an image, the key to recognize what is in it is to find the important parts or regions of interest. In other words, different parts of the image have different weights. Compared with an image, not only the regions in a video, but also the frames in a video play different roles. Sharma et al. [25] obtained great achievement with attention mechanism and they have proven that soft attention could gain the performance in CNN-RNN model for action recognition task. To obtain this information, we employ soft attention mechanism.
The relations in each frame are of great importance for recognizing action. To discovery these relations, we incorporate attention mechanism with LSTM. In this layer, hidden state h 1 t−1 generated from LSTM integrates input features X 1 t extracted from ConvNet. Then, we design a function that takes h 1 t−1 and X 1 t as the input and returns the unnormalized relevance score at time step t, which is shown in below: Once all relevance scores for feature sequence are computed, we normalize them to obtain the a 1 t,i : The K 2 vectors in the appearance feature cubes correspond to K 2 regions in the t-th frame and our model chooses to focus its attention on these K 2 regions. a 1 t,i is a softmax over K × K locations and denotes weight of the i-th region. At each time step, we update the parameters of attention model, and get the input x 1 t by the following function: After weighting the features from ConvNet, we get x 1 t and input it into LSTM, . The illustration of the above process at the time step t is shown in Figure 2. In Equation (4), the W T 1 , W 1 a , U 1 a and b 1 a are estimated together in model training procedure. " CNN " Figure 2. The illustration of attention process. a 1 t,i is generated from hidden state and input, and then refines the input of first layer of LSTM through attention weight.

"Attention-Again" Model
"Attention-again" model combines the neighboring frames with current frame to get the interesting part of the frame. In other words, it could obtain more information than tradition attention model. To incorporate "attention-again" model into our framework, we stack another LTSM layer. Then, we utilize the current state in bottom LSTM network, the previous hidden state in top LSTM network and the current input features to generate unnormalized relevance score at time step t: Similar to Equation (5), we normalize relevance score at time step and get a 2 t,i where h 1 t is hidden state generated from bottom layer at time t, and h 2 t−1 represents hidden state generated from top layer at time t − 1. In Equation (7), the W T 2 , W 2 pa , W 2 a , U 2 a and b 2 a are estimated together in model training procedure. After that, we obtain the input of top LSTM, which is determined by Equation (9): Then, we feed this x 2 t as features into top LSTM layer, . In Section 4.3.1, we discuss the number of layer with "attention-again" model.

Decode Network
After embedding attention mechanism into our framework, we then input the result of above LSTM 2 to decode network. The main methods consider all hidden state, and set so f tmax result as the final category, which is shown in below function: In our work, we only consider the hidden state from the last LSTM unit in top LSTM layer and acheive great performance. In addition, the loss will converge fluently compared to the main methods.

Datasets
To verify the effectiveness of our methods, we conduct experiments mainly on three public action recognition benchmark datasets, namely UCF-11 (YouTube Action), HMDB-51 and UCF-101, which are currently challenging annotated action recognition datasets.
UCF-11 dataset, also known as the YouTube Action dataset, consists of 1600 videos and 11 actions: basketball shooting, biking/cycling, diving, golf swinging, horse back riding, soccer juggling, swinging, tennis swinging, trampoline jumping, volleyball spiking, and walking with a dog. The clips have a frame rate of 29.97 fps and each video has only one action associated with it. We use 1120 videos for training and 480 videos for testing (70% for training and 30% for testing).
HMDB51 dataset is a large collection of realistic videos from various sources, including movies and web videos. It is composed of 6766 video clips from 51 action categories, with each category containing at least 100 clips. These clips are labeled with 51 classes of human actions such as Clap, Drink, Hug, Jump, Somersault, Throw and many others. This dataset is more challenging than others because it contains scenes with more complex backgrounds. It has similar scenes in different categories, and it has a small number of training videos. HMDB-51 have three split settings to separate the dataset into training and testing videos. Our experiments follow the original evaluation scheme, but only adopt the first training/testing split. In this split, each action class has 70 clips for training and 30 clips for testing.
UCF101 dataset is composed of about 13,320 realistic user-uploaded video clips and 101 action categories, with each category containing at least 100 clips. The database is particularly interesting because it gives the largest diversity in terms of actions and with the presence of large variations in camera motion, object appearance and pose, object scale, viewpoint, cluttered background, illumination conditions, etc, and it is the most challenging dataset to date. There are three splits for training and testing (70% training and 30% testing). We tested our model on the first training/testing split in the experiments. Classification accuracy is used as evaluation measure.

Training Details
As shown in Figure 1, we use a CNN-RNN network structure as in [25] to get our final prediction. It is believed that training a good deep convolutional neural network for videos understanding is more challenging. Similar to [25], we only train all the hyper-parameters in RNNs by inputing features extracted from pre-trained ConvNet.
In detail, the dimensionality of the LSTM hidden state, cell state, and the hidden layer were set to 1024. To avoid overfitting, we use dropout (similar to Srivastava et al. [24]) of 0.5 at all non-recurrent connections. After that, we employ Adam optimization algorithm (from Kingma et al. [46]) to train all models. Next step, the learning rate starts from 0.003, then we use exponential decay algorithm which was implied in TensorFlow toolbox and the decay rate is set to 0.9. We implement our model with TensorFlow [47], and our experiments are carried out on a workstation with a 4GHz Intel(R) Core (M) i7-4790 CPU, 128G RAM and an NVIDIA(R) GeForce 1080 GPU. In the process of training and testing, our model uses CUDA to accelerate the experimental process. For UCF-11 dataset, we set batch size as 16, while the batch size of both HMDB-51 and UCF-101 is set as 8.

Results and Analysis
In this part, we analyze our experiment in two aspects. (1) Quantitative analysis: We compare the result measured on three benchmark datasets with other state-of-art works under the same condition; (2) Qualitative analysis: We show some examples that illustrate the attention in the frame vividly.

The Effect of Different CNN Encoders
In the famous imageNet competition, ResNet have get great performance due to deep structure. Since action recognition is not the same as image classification, we verify the result of different CNN encoders. To date, there are four widely used CNN encoders, namely GoogLeNet, VGG, ResNet-101 and ResNet-152, to extract visual features. In this sub-experiment, we study the influence of different versions of CNN encoders on our framework. The experiments are conducted on RGB data on the UCF11 and first split of HMDB51 and UCF101 datasets. The results are shown in Table 1. These above networks are all pre-trained on imageNet dataset. We can easily find that, by taking ResNet-152 as the visual decoder, our method perform best with 91.2% on UCF11, 54.4% on HMDB51 and 87.7% on UCF101. The Effect of Every Component The baseline of these methods is a simple "encode-decode" model with CNN encoding the input videos and LSTM decoding features into categories. In [48], they compare baseline performance of LSTM, Attention-LSTM (ALSTM), ConvLSTM [49] and ConvALSTM. In Table 2, we list the performance of all these LSTM variants and our main component for action recognition. To conduct a fair comparison, we use VGG net as our encode network and conduct experiment on split 1 of UCF-101 and HMDB51. Our proposed method significantly outperforms the variants on each dataset. From the component analysis, we find that the three LSTM layers would perform worse than two LSTM layers, thus we use two LSTM layers as our architecture. Moreover, attention mechanism greatly improves the performance, and "attention-again" model gains the result of recognizing actions. Furthermore, in decode network, we only use the output of the last LSTM unit. To verify the performance of this small change, we compare with conventional methods which make softmax operation among the outputs of every LSTM unit. Then, we conduct experiments on UCF11 and use the same encode network (VGG). The results are shown in Figure 3. The loss is shown in Figure 3a, while Figure 3b represents accuracy. The red and blue lines in Figure 3 represent our method and softmax method, respectively. From Figure 3, it is easily concluded that our method is more fluent than softmax. More importantly, the accuracy of our method is higher than softmax. Therefore, this structure of decode network can be proven.

Comparison with LSTM-like architecture
In this sub-experiment, we list all state of the art methods that use a LSTM-like architecture for action recognition tasks by only using RGB data. To present the comparison completely and clearly, we elaborate on some factors, such as pre-training type and network architecture. In Table 3, our proposed method clearly performs the best among the LSTM-like architectures. Table 3. State-of-the-art comparison with LSTM-like architectures. Classification accuracy is used as evaluation measure which represented as recognition accuracies (%). The bold text represents the best result.

Method
Pre The performance of RGB data from HMDB51 dataset are not shown in [51].

Comparison with state of the art
In addition to the LSTM-like comparison in Table 3, another state of the art algorithm comparison is presented in Table 4, which shows the results from RGB data.
It is obvious that our "attention-again" model outperforms these above methods significantly in three datasets. Although there are some other methods that combine the RGB data and flow data together to improve the performance in benchmark datasets, our model performs competitively against RACNN models in its category (models using RGB features only).  Figure 4 shows the correct example in test on UCF-11 dataset, where we can see that our model mainly focuses on where the action is happening. Figure 4a is a horse-riding example; our model pays attention to the leg and head of horse, and the leg of human is also given attention. Figure 4b is soccer example; our model mainly attend to the person who is playing soccer as well as the ball. Figure 4c shows two man playing volleyball; our model focuses on the net and surrounding in addition to players. Wang et al. [53] suggest that the true meaning of an action lies in "'the change or transformation an action brings to the environment", e.g., kicking a ball. Thus, to recognize action, we not only need the feature from the human, but also need the feature of environment. In other words, we have to combine the foreground and background together to gain better performance. Fortunately, our model tackles this problem by using attention-again model. For example, in Figure 4b, our model not only attends to the person who is playing soccer, but also focuses on the playground. It is naturally believed that, when people judge whether the man in the image is playing soccer, the information of man with a ball is not enough; other contextual information such as playground plays a critical role. Similar to Figure 4b, Figure 4c shows that our model pays more attention to the net and surrounding information. Additionally, it is obvious to observe that the person in Figure 4b,c is not as highlighted as that in Figure 4a.
Simultaneously, we can also better understand failures of the model using the attention mechanism. Figure 5a shows the incorrect example (Ground truth: biking; Result:tennis), while Figure 5b is correct example of tennis and Figure 5c is correct example of biking. We could understand how the model performs the wrong result to some extent by reviewing Figure 5b,c. As Figure 5b shows, our model attempts to focus on the person playing tennis and some surroundings to recognize "'tennis", and the interesting regions are mostly located in the red blocks. In Figure 5c, our model pays attention to the position of bike, apart from region of where the person appears. Meanwhile, the interesting regions actually belong to not the red blocks but the yellow ones. However, we could find that the model only focuses on the person and cannot concentrate on where the bike appears in Figure 5a, thus our model incorrectly classifies the biking into tennis.
(a)  (c) Figure 5. The illustration of difference between incorrect example and correct example. The label of (a) is biking, but our model recognizes it as tennis, while (b) is correct example of tennis and (c) is correct example of biking. The red case labels the location of person and the yellow one represents the location of bike in image.

Conclusions
In this paper, we present a human-inspired model, namely "attention-again", for action recognition in videos. "Attention-again" model is a variant of the normal attention mechanism for classification and recognition of human activities in videos. Our framework consists of three major components: (1) convolutional feature extraction; (2) LSTM sequence modeling and attention model; and (3) the proposed "attention-again" model. We conduct several experiments on three benchmark datasets to demonstrate the great performance of model. From results shown in Section 4, we can easily conclude that our model outperform the state-of-the-art work with RGB data. Moreover, we visualize the region our model focuses on to understand how the model works. In future work, we will conduct our model using other state-of-the-art works and combine the advantages of different works to improve the accuracy of action recognition.