Memory Attention Networks for Skeleton-based Action Recognition

1 Introduction

3D skeleton-based human action recognition has recently attracted a lot of research interests due to its high-level representation and robustness to variations of viewpoints, appearances and surrounding distractions [Han et al., 2017; Presti and La Cascia, 2016; Ding and Fan, 2016]. It is motivated by the biological observations that human beings can recognize actions from just the motion of a few joints of the human body, even without appearance information [Johansson, 1973]. To describe human actions, conventional recognition approaches use relative joint coordinates to overlook the absolute movements of skeleton joints and thus gain partial view-invariant transformation. They include aligned spherical coordinates with person's direction [Xia et al., 2012], translated coordinates invariant to absolute position and orientation [Jiang et al., 2015], and flexible view invariant transform with principal components [Raptis et al., 2011].

1 The code will be made publicly available at https://github.com/memory-attention-networks.

∗ Baochang Zhang is the corresponding author.

† Chunyu Xie and Ce Li have equal contribution to the paper.

Skeleton sequences are time series of joint coordinate positions. To learn the temporal context of sequences, Recurrent Neural Networks (RNNs) [Li et al., 2017], Long ShortTerm Memory (LSTM) [Zhu et al., 2016], and Gated Recurrent Unit (GRU) [Cho et al., 2014], have been successfully applied to skeleton based action recognition. But it still challenging to cope with the complex spatio-temporal variations of skeleton joints caused by a number of factors, such as action speed, jitters, and surrounding distractions. To handle these variations, attention mechanism is introduced in [Liu et al., 2016; Liu et al., 2017a; Zhu et al., 2016; Song et al., 2017] to provide a robust recognition system. For instance, STA-LSTM [Song et al., 2017] allocates different attention weights for selecting key frames and discriminative joints within one frame. Similarly, GCA-LSTM [Liu et al., 2017a] selects the global informative joints from a sequence.

A few works exploit CNNs to solve the skeleton based action recognition problem. In [Ke et al., 2017], skeleton joints after being projected or encoded, are used as the input channels of CNNs, which causes temporal information loss during the conversion of 3D information (x, y, z joint coordinates) into 2D information (images). In [Liu et al., 2017b; Lea et al., 2016], skeleton joints in each frame are transformed and expressed as color heat maps in CNNs, where complex data preprocessing gives rise to the loss of distinct spatio-temporal information.

In this work, our goal is to bring these powerful tools (e.g. RNNs, CNNs and attention learning) under the same umbrella and develop an efficient framework to investigate a new hypothesis of "memory attention + convolution network" for skeleton based action recognition. We propose an end-to-end deep network architecture, termed as Memory Attention Networks (MANs), to perform temporal-then-spatial feature recalibration. It can leverage the state-of-the-art CNNs to enhance the spatio-temporal features [Zhang et al., 2016]. So far, CNNs particularly ResNets [He et al., 2016] have been the most popular tools due to the unique residual module. Inspired by it, we design our temporal-then-spatial recalibration scheme in MANs based on the residual module as shown in Fig. 1. By doing so, both the original input features and the attention information can be fully exploited by subsequent CNNs in a unified framework, leading to a comprehensive and effective feature representation.

Distinctions between this work and prior art. (1) The state-of-the-art attention network [Song et al., 2017] uses two LSTMs to model spatial and temporal attentions for each skeleton frame based on the input (frames) at time steps t and t-1. We also use RNN in TARM, but only to model the memory information of skeleton sequences. We design a new attention network to learn the attention weights and then make use of the learned temporal attention to recalibrate the original skeleton sequence in a residual module, which facilitates efficient learning of attention features. (2) The existing CNNs based skeleton action recognition methods [Ke et al., 2017; Liu et al., 2017b] involve complicated pre-processing. For example, [Ke et al., 2017] is based on clip generation (skeleton segmentation) and color images transformation; [Liu et al., 2017b] performs skeleton coordinate transform and generates images with visual enhancement as the input. However, our proposed MANs directly operate on the skeleton sequences without bells and whistles, enabling an end-to-end training of network.

Contributions. The contributions of this paper are threefold.

1. We propose an end-to-end framework of Memory Attention Networks (MANs) to demonstrate the powerful capacity of a new "memory attention + convolution network" scheme for modeling the complex spatio-temporal variations in skeleton joints. It is the first time that a "RNNs + CNNs" framework has been developed for skeleton-based action recognition.
2. A new attention learning method is presented based on the residual module. It recalibrates temporal features to pay more attention to informative skeleton frames.
3. MANs achieve the state-of-the-art results on four challenging datasets. We also perform extensive ablation study to show the effectiveness of each unit in MANs.

2 Memory Attention Networks

In this section, we elaborate the two modules: Temporal Attention Recalibration Module (TARM) and Spatio-Temporal Convolution Module (STCM) in MANs. Table 1 summarizes the notations used in this paper.

Temporal Attention Recalibration Module

（TARM）

Figure 2: The Residual Module (left), Temporal Attention Recalibration Module (middle), and BiGRU Memory Cell of TARM (right). TARM is designed based on the residual module as ˜X = X + F(X), which incoporates the input and recalibrated features in a unified framework. F(X) is the recalibrated feature, e.g., the output of our residual attention module, via F(X)=FM(X) ⊙FA(X).

2.1 Temporal Attention RecalibrationModule

The input skeleton data is a sequence of multi-frame 3D joint coordinates forming an action. Let O = {X,Y,Z} ∈ RT×N×3 , where X ∈ RT×N , Y ∈ RT×N , Z ∈ RT×N , denotes N joints along T frames with x, y, and z coordinates. For ease of explanation, X is chosen as an example to describe TARM.

As shown in Fig. 2, given a 2D matrix X, the learning of TARM pursuits a specific attention based on the BiGRU in memory cell to capture the temporal memory information across the input action sequence. More specifically, inspired by the original RM in ResNets, we construct TARM via identity mapping with transformation from X ∈ RT×N to ˜X ∈ RT×N to capture the richer temporal information, as

where F(X) is the recalibrated feature, e.g., the output of our residual attention module shown in Fig. 2, based on two branches: FM(X) and FA(X), which represent the memory information and attention weight, respectively.

where ⊙ denotes the element-wise multiplication. For simplicity, FM(X) and FA(X) are denoted as FM ∈ RT×K and FA ∈ RT×K , respectively. FA is the weight of FM to recalibrate temporal information. Obviously, for an action sequence, the importance of representative information in each frame is different, and only a few key frames containing important discriminative information deserves to be emphasized for action representation.

Calculating FM. We implement the memory cell via BiGRU. X ∈ RT×N is resized and updated by the output of a FC layer as X ← FC(X) ∈ RT×K , as shown at the top of Fig. 2. In a slight abuse of notation, we still denote the resized output as X. FM ∈ RT×K is the memory information made up of two directional combined hidden states in BiGRU, where K denotes the number of neuron units in BiGRU.

For simplicity, we still denote X ∈ RT×K as

where x(t) is a row vector of X to represent the sequence at tth frame as [x1(t), · · · , xK(t)], and t ∈ (1, ..., T).

As illustrated in Fig. 2, →GRU (x(t)) and ←GRU (x(t)) are the output hidden states of the forward GRU and backward GRU, respectively. Combining these bidirectional hidden states, the informative vector m(t) ∈ R1×K at the tth frame is denoted as

Finally, all the outputs are concatenated across T frames, and the memory information of skeleton joints is represented as

where FM summarizes the memory information in BiGRU for the skeleton joints across the sequence.

Calculating FA. To recalibrate the memory information FM, the attention weight FA is exploited as shown in Eq. (2).

In Fig. 2, following X ∈ RT×K , our recalibration scheme can capture global frame-wise dependence across T frames. We first aggregate each row vector of X in Eq. (3) by the average pooling operation to produce a T × 1 vector as

We then duplicate it with K copies as

In the right branch of TARM shown in Fig. 2, the attention mechanism is represented by a bottleneck with the two FC layers providing the non-linear interaction between frames. We introduce a dimensionality-reduction layer with parameters W1 and a ratio factor α (empirically set to be 16), followed by a ReLU function. And then, we introduce a dimensionality-increasing layer with parameters W2 and a sigmoid activation function. The dimensionality-reduction and dimensionality-increasing processing can be considered as the denoising and excitation operations respectively, and thus enhance the feature discriminability. Finally, the output of the attention branch FA is calculated as

where W1 ∈ R Tα ×T and W2 ∈ RT× Tα . To simplify the notation, the bias terms in Eq. (8) are omitted. θ (·) refers to the ReLU function, and σ (·) denotes the sigmoid function. Finally, F(X) is obtained by the element-wise multiplication of FM and FA.

Furthermore, to calculate the output feature map of TARM, ˜X, F(X) ∈ RT×K is resized by a FC layer as F(X) ← FC(F(X)) ∈ RT×N , shown at the bottom of Fig. 2. As a result, the final F(X) describes the temporal information of the entire skeleton sequence. Similar to RM in ResNets, FM and FA in TARM can be jointly learned during training. In a similar way, we can obtain ˜Y and ˜Z based on Y and Z in their corresponding TARM, and ˜O = { ˜X, ˜Y, ˜Z} ∈ RT×N×3 .

2.2 Spatio-Temporal ConvolutionModule

Conventional attention methods in skeleton action recognition are limited by the modeling capacity of RNNs [Liu et al., 2016; Liu et al., 2017a]. STCM is introduced based on CNNs to extract the enhanced spatio-temporal features from the output ( ˜X, ˜Y and ˜Z) of the TARM. By leveraging the robustness to deformation of CNNs, STCM further extracts high-level feature representations to better cope with spatiotemporal variations of skeleton joints.

In principal, any CNNs can be used in STCM, e.g., DenseNets [Huang et al., 2017]. ˜O ∈ RT×N×3 denotes the output of TARMs and also the input to STCM, and FC denotes the output of STCM for the softmax classifier. For example, in Fig. 1, the BN-ReLU-Conv blocks in STCM are used to interpret the high-level spatial structures of skeleton joints as

Afterwards, FC is fed to a softmax classifier to predict the class label as

whereWC and ˆy denote the weights in the softmax layer and the predicted action label, respectively. The cross-entropy loss function [Goodfellow et al., 2016] is adopted to measure the difference between the true class label y and the prediction result ˆy.

3 Experiments

The proposed MANs are evaluated on four public skeleton action datasets: NTU RGB+D [Shahroudy et al., 2016], HDM05 [M¨uller et al., 2005], SYSU-3D [Hu et al., 2015] and UT-Kinect [Xia et al., 2012].

3.1 Implementation Details

For all the datasets, the matrices ( ˜X, ˜Y and ˜Z) are generated with all the frames of a skeleton sequence. We use two different scales for the three input matrices of each sequence, i.e., 224 × 224 and 50 × 50, respectively. For the large scale, the number of hidden units of BiGRU in TARM is set to 2 × 128 (K = 128), where 2 indicates bidirectional GRU, 128 is the number of neurons. DenseNet-161 [Huang et al., 2017] is used in STCM, leading to MANs (DenseNet-161). For the small scale, we set the hidden units of BiGRU to 2 × 64, and stack multiple BN-ReLU-Conv blocks as STCM, resulting in MANs-n (e.g. n = 9) where n is the number of BN-ReLUConv blocks. The architectures of various MANs are illustrated in Table 2.

MANs are trained using the stochastic gradient descent algorithm, and the learning rate, decay, and momentum, are respectively set to 0.1, 0, and 0.9. The mini-batches of samples on NTU RGB+D, HDM05, SYSU-3D, and UT-Kinect are constructed by randomly sampling 40, 20, 8, and 8 samples from the training sets, respectively. The training stops after 100 epochs except for NTU RGB+D after 50 epochs. For a fair comparison, the performance of MANs on each dataset is compared with existing methods using the same evaluation protocol. All experiments are performed based on Keras 2 with Tensorflow backend using two NVIDIA Titan X Pascal GPUs.

3.2 Experiment Analysis

Ablation study. We conduct extensive ablation study of different units in MANs with the following settings: (A) STCM 3 (i.e. applying CNN on the original skeleton images); (B) MANs (no attention)-MANs without attention in TARM (i.e. no FA); (C) MANs (other temporal attention)-MANs use the temporal attention scheme in [Song et al., 2017] for the TARM. Table 3 shows the results of different architectures on the NTU RGB+D dataset. Note that the STCM of MANs uses DenseNet-161 for the input of 224 × 224, and uses stacked BN-ReLU-Conv blocks with 9 layers for the input of 50 × 50 in Table 3, respectively. Comparing with our full MANs model, we have these findings. (1) Setting A yileds much lower performance, indicating the importance of temporal information modeling for skeleton. (2) Setting B reveals the effectiveness of the proposed attention mechanism (i.e. learning FA). (3) Setting C substitutes the attention scheme in [Song et al., 2017] for our residual attention module in MANs. The recognition accuracy is lower than that of our MANs, which again validates the superiority of our residual attention learning approach.

2 http://keras.io

3 Here STCM takes [X,X,Z] as image input to perform classification via CNNs.

Learning convergence. We plot the training error and testing error curves of the four networks on a same trail of NTU RGB+D dataset in Fig. 3, including STCM-9, MANs-9 (no attention), MANs-9 (other temporal attention), and MANs9. We can observe that MANs-9 in solid line converges at epoch #26 for training and obtains the best error rate of 16.99% for testing. For training error, STCM-9 stops decreasing at epoch #30, MANs-9 (no attention) converges at epoch #30, and MANs-9 (other temporal attention) converges at epoch #40. These curves show that MANs obviously converge faster and gain better performance than others, which proves that the "memory attention + convolution network" scheme in MANs can be used to improve the modeling ability of CNNs. MANs-9 converges much faster than MANs-9 (other temporal attention) (epoch #26 v.s. #40), due to the novel residual attention module which takes the input and attention information into account in the same framework. More specifically, the residual attention module not only uses

(a) Training error v.s. epoch

the temporal attention recalibrated information, but also delivers the spatial structure information of the original input by the identity shortcut.

Various CNNs in STCM. The number of stacked layers of CNNs in STCM, i.e., multiple BN-ReLU-Conv blocks used in MANs-9, MANs-33 and MANs-61, is evaluated in terms of the recognition accuracy on all the datasets (Table 4). We also include MANs (DenseNet-161) in the same table. The deeper CNNs have more learnable parameters than the shallower ones. We note that MANs-9 with a similar parameter amount as Deep LSTM [Zhu et al., 2016] has much better performance than the state-of-the-arts. This reveals that our method is more effective if the network complexity should be considered. Interestingly, MANs-9 mostly achieve better performance than other deeper models, which is probably due to its compactness. However, the very deep MANs (DenseNet161) obtain the best result on the challenging NTU RGB+D dataset in cross-view setting. It is worth noting that because of the "memory attention + convolution network" scheme of MANs, it is quite flexible to deploy different parameter amount in MANs by adjusting the number of CNN layers in STCM to balance between performance and network complexity.

3.3 Results and Comparisons

We show the performance comparison of various MANs architectures with other state-of-the-art approaches in Table 4 for the four datasets, respectively. Note that all our MANs under different parameter amounts are able to achieve better performance than the state-of-the-art RNNs-based approaches (e.g. VA-LSTM [Zhang et al., 2017], GCA-LSTM [Liu et al., 2017a]) and the state-of-the-art CNNs-based approaches (e.g. Clips+CNN+MTLN [Ke et al., 2017]), demonstrating the superiority of our "memory attention + convolution network" architecture.

For the NTU RGB+D dataset, MANs perform significantly better than others in both the cross-subject and crossview protocols. Comparing to other methods, MANs-9 increase the accuracy by 3.44% for cross-subject evaluation, and MANs (DenseNet-161) lead to a significant 5.62% improvement on this largest dataset in cross-view evaluation, which demonstrate that MANs can learn more discriminative spatio-temporal features to alleviate the spatial and temporal variations in skeleton joints.

For the HDM05 dataset, our MANs-33 achieves better result than the state-of-the-art multi-layer RNNs-based models. For the SYSU-3D dataset, four accuracies of our MANs are higher than all the RNNs-based methods [Hu et al., 2015; Liu et al., 2016; Liu et al., 2017a; Zhang et al., 2017]; especially MANs-61 outperform the previous best approach GCA-LSTM by 9.03%. For the UT-Kinect datasest, MANs9 and MANs-33 achieve a 100% accuracy, with 1.0% improvement in comparison with the state-of-the-art GCALSTM. The experiments show that compared with the existing RNNs-based methods, the deployed residual module with temporal attention and the temporal-then-spatial scheme can effectively improve the modeling ability of RNNs.

4 Conclusion

In this paper, we propose an end-to-end framework, termed Memory Attention Networks (MANs), to enhance the spatiotemporal features for skeleton-based action recognition. In MANs, TARM is designed to recalibrate the temporal attention to skeleton frames in action sequences, and STCM further models the spatial structure and temporal dependence of the skeleton sequence by leveraging the powerful CNNs. Through the unified framework, MANs significantly boost the performance for skeleton-based action recognition. The extensive experiments validate the superiority of MANs, which consistently perform the best on four benchmark datasets and contribute new state-of-the-art results.

Acknowledgements

The work was supported by the Natural Science Foundation of China under Contract 61672079, 61601466, and 61473086, the Open Projects Program of National Laboratory of Pattern Recognition, and Shenzhen Peacock Plan KQTD2016112515134654. Baochang Zhang is also with Shenzhen Academy of Aerospace Technology, Shenzhen, China.