Multi-Scale Spatial-Temporal Integration Convolutional Tube for Human Action Recognition

1 Introduction

With the rapid development of various video platforms in the social network, video is becoming a popular communication medium among internet users. This has encouraged the development of advanced techniques for a variety of video understanding applications. More specifically, one of the most fundamental tasks that ensures the success of these technological advances is human action recognition. Human action recognition aims to recognize actions by the visual appearance and motion dynamics of the involved humans and objects in video sequences. Recently, the Convolutional Neural Networks (CNNs) demonstrate the high capability of learning visual representation in image domain and the way that extends CNNs into video domain is the main proposal for the action recognition task.

∗Equal contribution

†Corresponding author

Video actually can be seen as a sequence of images in the dimension of time. A good model should be able to extract not only the spatial appearance in images, but the dynamic motion change over time. Traditional 2D-CNN base methods [Simonyan and Zisserman, 2014; Donahue et al., 2015] neglected the joint exploration of spatial appearance and temporal motion, which could offer a comprehensive representation of videos and thus enhance the accuracy of action recognition. For most 3D-CNN based methods [Tran et al., 2015; 2018; Carreira and Zisserman, 2017; Wang et al., 2018b], they more or less integrated spatial and temporal information. However, with the addition of a new dimension, the parameters and computational cost of the 3D-CNN based models are always extremely high compared to the 2D-CNN based models. The P3D [Qiu et al., 2017] took the lead in separating the 3D convolution into two separate convolutions, i.e., a 2D spatial convolution plus a 1D temporal convolution, and thus significantly reduced the model size. Nevertheless, this kind of method still ignored the correlation of spatial appearance and temporal motion.

Recently, the multi-scale representations [Szegedy et al., 2017; Gao et al., 2019; Liu et al., 2016] are of critical importance to a number of vision recognition tasks. For human action recognition task, the humans and objects may appear with different spatial sizes in images, and the actions may also last for different lengths of time in videos. However, directly obtaining layer-wise multi-scale spatial-temporal features requires feature extractors to use a large range of receptive fields, which will add a large amount of extra computational cost. In addition, the interaction between the feature maps at different scales can guide themselves to pay attention to informative features rather than useless features, since the contextual information of an action may occupy a much larger area than the action itself.

In this paper, we propose a Multi-Scale Spatial-Temporal Integration Convolutional Tube (MSTI) aiming towards robust and accurate human action recognition tasks. The MSTI tube generates multi-scale spatial appearance and temporal motion through multi-group convolution, and then applies feature recalibration by integrating multi-scale spatial and temporal features to obtain effective spatial-temporal features, simultaneously reducing the computational cost. Specifically, the MSTI tube consists of multi-scale convolution block and cross-scale attention weighted blocks. An illustration of our MSTI tube is shown in Fig.1. The multiscale convolution block divides the input tensor into several groups, and each group has their own convolutional filters. For the first group, it's output feature maps are directly calculated by it's input feature maps and filters. For the other groups, the previous group's output feature maps are sent to the current group's filters along with the current group's input feature maps. The cross-scale attention weighted blocks integrate multi-scale spatial and temporal features, aiming at selectively emphasizing informative spatial-temporal features and suppressing less useful ones. In the spatial branch, the cross-scale attention weighted block takes the previous group's spatial feature maps, the current group's spatial and temporal feature maps as inputs, and then generates optimized spatial feature maps; In the temporal branch, it takes the previous group's temporal feature maps, the current group's temporal and spatial feature maps as inputs, and then generates optimized temporal feature maps.

The main contribution of this work can be briefly summarized as: 1) we design the multi-scale convolution block to capture multi-scale representations on both spatial and temporal domain; 2) we design the cross-scale attention weighted blocks which integrate multi-scale spatial and temporal features, aiming to perform feature recalibration by selectively emphasizing informative spatial-temporal features and suppressing less useful ones; 3) a deep network with low computational cost, MSTI-Net, is put forward for learning robust and accurate video representation. Experiment results show that our MSTI-Net outperforms other methods on three challenging action recognition benchmarks, Kinetics-400, UCF101 and HMDB-51 with lower computational cost.

2 RelatedWork

With the rapid development of convolutional neural networks in the field of image, the video field is becoming a more and more popular field people try to expand into. According to the types of convolutions used in features learning, existing action recognition works can be briefly divided into two categories: 2D CNN and 3D CNN based methods.

2DCNNbased. Karpathy et al. [Karpathy et al., 2014] proposed a "slow fusion" model which took the lead in fusing temporal information into 2D CNNs. The model firstly extended temporal connectivity of all convolutional layers and then computed activation through temporal and spatial convolutions. The two-stream structure proposed by [Simonyan and Zisserman, 2014] is one of the influential approaches which directly used two 2D CNNs to capture spatial and temporal information respectively from RGB frames and stacked optical flows, improving video recognition accuracy. Following this idea, several studies have been presented to fuse these two networks over the appearance and motion, e.g. the STResNet [Feichtenhofer et al., 2016] and the temporal segment networks [Wang et al., 2016]. LRCN [Donahue et al., 2015] tried to explore the possibility of combining LSTM networks with frame-level features of 2D CNNs to explicitly model spatial-temporal relationships. Recently, the multi-scale representations which are of great importance to various vision tasks have been widely used in a number of networks, such as Inception-Nets [Szegedy et al., 2017] and Res2Net [Gao et al., 2019]. Meanwhile, at the channel level, the SE-Net [Hu et al., 2018] extracted different channels' attention to further improve the quality of representations produced by a 2D network, which achieved impressed results on image classification.

3DCNNbased. The 3DConvolutional Networks were first presented for learning video representations over 16-frame video clips in the context of large-scale supervised video datasets [Tran et al., 2015]. Compared to 2D kernels which merely model the spatial information, 3D convolution kernels have the capability of modeling more complex relations between appearance and motion. The Res3D [Tran et al., 2017] made one step further by taking the advantage of residual connections to facilitate training. Similarly, I3D [Carreira and Zisserman, 2017] was proposed to use the inception network [Szegedy et al., 2017] as backbone rather than residual networks to learn video representations. Many architectures were also proposed to improve 3D convolution [Tran et al., 2018; Zhou et al., 2018; He et al., 2018; Wang et al., 2018b]. However, all of these methods demanded more than an order of magnitude computational cost than their 2D competitors. This made them difficult to train and apply to practical applications. To overcome the limitation of 3D CNN and decrease the number of parameters, the P3D [Qiu et al., 2017] decomposed a 3D convolution kernel into a 2D spatial kernel and a 1D temporal kernel to reduce the computations of a 3D convolutional layer and achieved better precision at the same time. Wu et al. [Wu et al., 2019a] further optimized this kind of structure, and proposed a mutually reinforced spatio-temporal convolutional tube (MRST) to learn the correlation between spatial and temporal features. Based on the depth-wise separable convolutions idea, the work [Wu et al., 2019b] proposed depth-wise separable 3D convolution networks, which commendably simplified the large inputs tensor.

3 MSTI Tube and DeepMSTI Network

3.1 MSTI Tube

The multi-scale spatial-temporal integration convolutional tube (MSTI) applies the bottleneck structure, as shown in Fig.1, which employs two 1 × 1 × 1 convolutional layers at both ends of the path to reduce and restore the channel dimensions respectively, decreasing the overall computational cost. In this section, we will first introduce concrete details of the composition of the MSTI tube, i.e., the multi-scale convolution block and the spatial and temporal cross-scale attention weighted blocks. We then present our robust and efficient deep network, MSTI-Net for human action recognition.

Multi-Scale Convolution Block

In the multi-scale convolution block, we first evenly slice the 3D input feature maps X ∈ RL×H×W×C into four groups, denoted by Xi ∈ RL×H×Wט C , where i ∈ {1, 2, 3, 4}, and L,H,W, ˜ C refer to the length, height, width and the number of group channels, respectively.

In the MSTI-spatial branch, each group Xi has a corresponding 1 × 3 × 3 spatial convolution, except that the first group X1 is followed by a 1×1×1 spatial convolution. The corresponding spatial convolution of each group are denoted by Ksi , and the outputs of each spatial convolution are named Si. The whole multi-scale spatial convolution architecture presents a stepped structure, as shown in Fig.2(a). The output Si can be written as:

From the above formula, we notice that each time the spatial features split Si goes through a 1×3×3 spatial convolutional kernel, the output result can have a larger spatial receptive field than Si, increasing by 2 in both the height and width dimensions. In this way, we get four different spatial receptive fields, i.e., 1 × 1 × 1, 1 × 3 × 3, 1 × 5 × 5 and 1 × 7 × 7. With these multi-scale spatial features, we can learn a more discriminative spatial representation.

Similar to the MSTI-spatial branch, in the MSTI-temporal branch, each input group Xi has a corresponding 3 × 1 × 1 temporal convolution, except that the first group X1 is followed by a 1 × 1 × 1 temporal convolution. We denote the corresponding temporal convolution of each group by Kti , and named each temporal convolution's outputs Ti, as shown in Fig.2(b). The output Ti can be calculated by:

Each time the temporal features split Ti goes through a 3 × 1 × 1 temporal convolutional kernel, the output's temporal receptive field increases by 2 in the length dimensions. Therefore, we obtain multi-scale temporal representation with four different temporal receptive fields, i.e., 1 × 1 × 1, 3 × 1 × 1, 5 × 1 × 1 and 7 × 1 × 1.

In addition, the multi-scale (MS) convolution can also be applied to reduce the impact of the large Cin or Cout on computational complexity. We present the parameters of the 3D convolution and our MS convolution in terms of formulas, which can be shown as follows:

where d and k×k refer to the temporal and spatial kernel size, respectively. We can see that compared to the original 3D convolution, the MS convolution can reduce the number of parameters by a factor of N, and in the meantime keeping the height, width as well as the length of feature maps unchanged. We set N = 4 in our multi-scale convolution.

Cross-Scale Attention Weighted Blocks

The spatial and temporal cross-scale attention weighted (CAW) blocks in the MSTI tube are applied after the multiscale convolution block. The detailed architecture of spatial and temporal CAW blocks are illustrated with Fig.2(c). In the spatial and temporal CAW blocks, we integrate spatial feature maps, temporal feature maps and the previousscale feature maps to further optimize spatial and temporal features, respectively, selectively emphasizing informative spatial-temporal features and suppressing less useful ones.

We make a visualization of spatial and temporal feature maps at different scales, as shown in Fig.3. we find that the spatial feature maps (middle column) mainly concentrate on the objects with surrounds, such as the human appearance and the backgrounds, and the temporal feature maps (right column) mainly concentrate on the objects which are in continuous motion, such as the human's body and legs. Integrating spatial and temporal feature maps can give a guideline to perform feature recalibration, which can pay more attention to the humans and objects' spatial appearance and temporal motion rather than the backgrounds. In addition, compared to the large-scale feature maps (the second row) which have larger receptive fields and contain more spatial-temporal information, the resolution of the small-scale feature maps (the first row) is much finer, which can guide the attention of largescale resolution areas. Besides, in terms of the multi-scale convolution structure, the previous group's outputs are a part of the current group's inputs. Thus, the small-scale feature maps is instructive to the large-scale feature maps.

We suppose each group's multi-scale spatial and temporal convolution output tensors have the same size Si, Ti ∈ RL×H×W× ˜ C . Then we can rewrite them as Si = [si1, si2, . . . , si ˜ C ] and Ti = [ti1, ti2, . . . , ti ˜ C ], where sic, tic refer to the features of the c-th channel in Si and Ti, respectively. We first utilize three 1 × 1 × 1 convolutional layers to integrate the spatial-temporal feature maps of the current group and the optimized feature maps of the previous group, except that in the first group, we only apply two 1×1×1 convolutional layers to integrate it's spatial and temporal feature maps. In the spatial and temporal CAW blocks, we calculate the i-th group integrated features Ui as follows:

where USi and UTi refer to the i-th group integrated spatial and temporal features, respectively; W∗SS , WSS, WTS refer to the parameters of the three 1 × 1 × 1 convolutional layers in spatial CAW block, and W∗TT , WTT , WST refer to the parameters of the three 1 × 1 × 1 convolutional layers in temporal CAW block. We can further rewrite Ui as Ui = [ui1, ui2, . . . , ui ˜ C ].

We then use a global average pooling operation to generate channel-wise statistics. Formally, a statistic zi ∈ R ˜ C is generated by shrinking the i-th group integrated features Ui through its spatial-temporal dimensions L×H×W, therefore the c-th element of zi is calculated by:

To make a good use of the information aggregated in the global average pooling operation, we follow it with an excitation operation aiming to fully capture channel-wise dependencies. The excitation function must firstly be capable of learning a nonlinear interaction between channels, and secondly it must learn a non-mutually-exclusive relationship since we would like to ensure that multiple channels are allowed to be emphasized. Thus, we apply the fully connected layer and the sigmoid function to make the excitation operation, and the i-th group attention output is denoted by ai. The function can be shown as follows:

where σ refers to the sigmoid function, and W ∈ R ˜ C× ˜ C denotes the weights of the fully connected layer.

The final outputs of the spatial and temporal CAW blocks S∗i and T∗i are obtained by re-scaling multi-scale convolution outputs Si and Ti with the activation ai:

where S∗i = [s∗i1 , s∗i2 , . . . , s∗i ˜ C ], T∗i = [t∗i1 , t∗i2 , . . . , t∗i ˜ C ] and the weighted function refers to channel-wise multiplication between the output attention and the spatial-temporal feature maps.

Finally, we concatenate each group's optimized spatial and temporal features S∗i and T∗i together and calculate the final outputs of our MSTI tube X∗ as follows:

3.2 DeepMSTI Network

We propose an efficient and effective MSTI-Net based on the ResNet-50 structure. The proposed MSTI-Net has 50 layers, which contains an initial convolutional layer (conv1), a max-pooling layer, four convolutional residual blocks (conv2-conv5) and a fully connected layer. The main idea of the residual block is to learn the additive residual function with reference to the unit inputs which is realized through a shortcut connection, instead of directly learning unreferenced non-linear functions [He et al., 2016]. The repeat times, kernel size, strides and output size of each convolutional block are all shown in Table 1. In the conv1 layer which is mainly proposed to learn rough spatial-temporal features, we apply 3 × 7 × 7 as kernel. And in the other convolutional blocks, we apply the MSTI tube as kernel. We don't apply the temporal pooling in the last three convolutional blocks in order to ensure the effectiveness of multi-scale convolution and multiscale spatial-temporal information interaction.

4 Experiments

4.1 Datasets and Implementation Details

Datasets. We use threewidely-used and challenging benchmarks, i.e. Kinetics-400 [Kay et al., 2017], UCF-101 [Soomro et al., 2012], and HMDB-51 [Kuehne et al., 2013] in the experiments. The large-scale Kinetics-400 dataset consists of about 300, 000 videos from 400 action categories. The UCF-101 dataset is composed by 101 action categories and 13,320 manually labeled video clips in total. The HMDB-51 dataset is collected from various sources, e.g. movies and web videos. It is composed by 51 categories and 6,849 labeled video clips in total. Both UCF-101 and HMDB-51 consists of three training/test splits provided by the datasets organizers. We report the accuracy by averaging over all 3 splits.

Implementation details. Our data augmentation includes random clipping on both spatial dimension (by firstly resizing the smaller video side to 256 pixels, then randomly cropping a 224 × 224 patch) and temporal dimension (by randomly picking the starting frame among those early enough to guarantee a desired number of frames). To obtain the video predictions, we average clip predictions uniformly sampled from the long video sequence. We apply batch normalization and ReLU nonlinearities [Nair and Hinton, 2010] to all convolutional layers in our proposed network. We use the Adam Gradient Descent optimizer with an initial learning rate of 1e-4 to train the MSTI-related networks from scratch. The drop out ratio is set to 0.5 and the weight decay rate is set to 5e-5 . The gradient descent optimizer has the 1e-5 initial learning rate, and it is adopted with a momentum of 0.9 to train our MSTI-Net initialized with the Kinetics-400 and ImageNet-1k pre-trained model. To prevent over-fitting, we further employ a higher drop out ratio of 0.9 and a weight decay rate of 5e-4 .

4.2 Ablation Study

To demonstrate the effectiveness of each component of our proposed MSTI tube, we conduct a series of ablation experiments on UCF-101 and HMDB-51 datasets. We choose the P3D-B [Qiu et al., 2017] architecture as our baseline, which is the only parallel structure in all three P3Ds. All architectures use the same network backbone (with 8 convolutional layers, 5 max-pooling layers, and 2 fully connected layers) and the same 3D input size for a fair comparison. All architectures were pre-trained on the Kinetics-400 dataset.

Table 2 provides the comparison results in terms of the Top-1 classification accuracy on both UCF-101 and HMDB51 datasets. We notice that compared to the P3D-B baseline the multi-scale temporal convolution improves accuracy by 1.5% on UCF-101 and 3.4% on HMDB-51, and the multi-scale spatial convolution can improve the performance by 2.7% and 3.9% on UCF-101 and HMDB-51, respectively. Applying multi-scale convolution on both spatial and temporal dimensions can further improve the performance compared to the multi-scale temporal convolution and multi-scale spatial convolution, achieving 90.7% accuracy on UCF-101 and 67.3% accuracy on HMDB-51, which demonstrates that the multi-scale convolution is effective and can greatly improve performance. We also observe our final MSTI tube which contains the spatial and temporal CAW blocks gets the best performance, obtaining 92.8% and 70.4% accuracy on UCF-101 and HMDB-51, respectively. This can demonstrate the importance of integrating multi-scale spatial and temporal features, which aims to perform feature recalibration by selectively emphasizing informative spatial-temporal features and suppressing less useful ones. Overall, we verify the effectiveness of two proposed blocks (multi-scale convolution block and cross-scale attention weighted blocks) in our proposed MSTI tube and we can see that the MSTI tube gets a tremendous increase compared to the baseline P3D-B.

4.3 Comparison to the State-of-the-Art Methods

We further demonstrate the advances of our proposed MSTINet in comparison with state-of-the-art methods for human action recognition. For fair comparison, all methods use only RGB inputs. The detailed structure of our MSTI-Net is shown in Table 1. We uniformly sample 20 clips per video and average these clip predictions to obtain the final video prediction. The results on Kinetics-400, UCF-101 and HMDB-51 are shown in Tables 3 and 4, respectively.

Results on Kinetics-400. Table 3 shows the performance comparison of our proposed MSTI-Net (pre-trained on ImageNet-1k) against ten state-of-the-art methods in terms of Top-1 classification accuracy on Kinetics-400. The proposed MSTI-Net achieves 76.1% Top-1 classification accuracy, and the total number of parameters is 16.4M and the FLOPs is 46.3G. We can see that our MSTI-Net surpasses existing methods, improving the baseline C3D network[Tran et al., 2015] by 20.0% at Top-1 classification accuracy. Moreover, our MSTI-Net improves the second best compared method SlowFast[Feichtenhofer et al., 2019] by 0.5% in terms of Top-1 classification accuracy. In addition to this, the total number of parameters and FLOPs of our MSTI-Net are much fewer than those of most methods in the table. Compared to the C3D baseline, the MSTI-Net has only 1/5 of the parameters and 1/6 of the FLOPs. In short, the comparison indicates that our MSTI-Net can learn more effective spatial-temporal features much more efficiently.

Results on UCF-101 and HMDB-51. We also evaluate the fine-tuning MSTI-Net (pre-trained on ImageNet-1k and Kinetics-400) on UCF-101 and HMDB-51 datasets to investigate the generality and robustness. From Table 4, we can observe that our proposed MSTI-Net outperforms all the existing state-of-the-art methods with only RGB inputs on both UCF-101 and HMDB-51, which obtains 97.1% Top-1 classification accuracy on UCF-101 and 76.8% Top-1 classification accuracy on HMDB-51. Compared to the second best method R(2+1)D-RGB[Tran et al., 2018] on UCF-101, our MSTINet improves 0.3% Top-1 classification accuracy. Compared to the second best method MRST-Net[Wu et al., 2019a] on HMDB-51, our MSTI-Net gets 1.4% Top-1 classification accuracy improvement.

5 Conclusion

In this work, we address the problem of building highly efficient deep neural networks for human action recognition from the perspectives of generating multi-scale representations and integrating multi-scale spatial-temporal features. We propose a novel Multi-Scale Spatial-Temporal Integration Convolutional Tube in which the multi-scale convolution block generates multi-scale spatial appearance and temporal motion, and the cross-scale attention weighted blocks perform feature recalibration by integrating multi-scale spatial and temporal features. Benefiting from the two blocks, our MSTI-Net requires significantly less computational resources yet achieving the state-of-the-art action recognition accuracy.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant 2017YFB1300201, the National Natural Science Foundation of China (NSFC) under Grants U19B2038, 61620106009 and 61725203 as well as the Fundamental Research Funds for the Central Universities under Grant WK2100100030.