{"type":"ScholarlyArticle","title":"Multi-Scale Spatial-Temporal Integration Convolutional Tube for Human Action Recognition","authors":[{"name":"Haoze Wu","slug":"haoze-wu","given":"Haoze","family":"Wu","affiliation":"University of Science and Technology of China"},{"name":"Jiawei Liu","slug":"jiawei-liu","given":"Jiawei","family":"Liu","affiliation":"University of Science and Technology of China"},{"name":"Xierong Zhu","slug":"xierong-zhu","given":"Xierong","family":"Zhu","affiliation":"University of Science and Technology of China"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"Hefei University of Technology"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"University of Science and Technology of China"}],"abstract":"Applying multi-scale representations leads to consistent performance improvements on a wide range of image recognition tasks. However, with the addition of the temporal dimension in video domain, directly obtaining layer-wise multi-scale spatial-temporal features will add a lot extra computational cost. In this work, we propose a novel and efficient Multi-Scale Spatial-Temporal Integration Convolutional Tube (MSTI) aiming at achieving accurate recognition of actions with lower computational cost. It firstly extracts multi-scale spatial and temporal features through the multi-scale convolution block. Considering the interaction of different-scales representations and the interaction of spatial appearance and temporal motion, we employ the cross-scale attention weighted blocks to perform feature recalibration by integrating multi-scale spatial and temporal features. An end-to-end deep network, MSTI-Net, is also presented based on the proposed MSTI tube for human action recognition. Extensive experimental results show that our MSTI-Net significantly boosts the performance of existing convolution networks and achieves state-of-the-art accuracy on three challenging benchmarks, i.e., UCF-101, HMDB-51 and Kinetics-400, with much fewer parameters and FLOPs.","doi":"10.24963/ijcai.2020/105","url":"https://ijcai.org/proceedings/2020/105","pdf":"https://ijcai.org/proceedings/2020/0105.pdf","markdown":"https://ijcai.org/proceedings/2020/105.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/105","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":105,"pages":{"first":753,"last":759},"keywords":["Computer Vision: Action Recognition","Computer Vision: 2D and 3D Computer Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24022/file_24022.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"Method Backbone Input×clips number Kinetics-400 #Params FLOPs C3D [Tran et al., 2015] - [16 × 3 × 112 × 112] × 1 56.1% 79.0M 296.7G LRCN [Donahue et al., 2015] - [25 × 3 × 224 × 224] × 1 63.3% 9.0M 41.5G ARTNet [Wang et al., 2018a] ResNet18 [16 × 3 × 112 × 112] × 25 69.2% 35.2M","title":"Method Backbone Input×clips number Kinetics-400 #Params FLOPs C3D [Tran","authors":null,"year":"2015","venue":"ResNet","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"7G I3D-RGB [Carreira and Zisserman, 2017] BN-Inception [All × 3 × 256 × 256] × 1 71.1% 12.7M 544.4G StNet [He et al., 2018] ResNet101 [25 × 15 × 256 × 256] × 1 71.4% 52.2M 310.5G R(2+1)D-RGB [Tran et al., 2018] ResNet34 [32 × 3 × 112 × 112] × 10 72.0% 63.8M 152.4G S3D [Xie et al., 2018] BN-Inception [All × 3 × 224 × 224] × 1 72.2% 8.8M 518.6G MRST-Net [Wu et al., 2019a] ResNet101 [16 × 3 × 224 × 224] × 20 74.1% 31.7M","title":"BN-Inception [All × 3 × 256 × 256","authors":"G I3d-Rgb [ Carreira, Zisserman ; He","year":"2017","venue":"ResNet","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"6G CFST [Wu et al., 2019b] ResNet34 [16 × 3 × 224 × 224] × 50 74.6% 12.9M","title":"ResNet","authors":"G Cfst [wu","year":"2019","venue":"ResNet","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"1G Nonlocal-I3D [Wang et al., 2018b] ResNet50 [32 × 3 × 224 × 224] × 10 74.9% 35.3M 163.3G SlowFast [Feichtenhofer et al., 2019] ResNet50 [64 × 3 × 224 × 224] × 10 75.6% 32.9M 36.1G MSTI(ours) ResNet50 [16 × 3 × 224 × 224] × 20 76.1% 16.4M","title":"1G Nonlocal","authors":"I3d [wang","year":"2018","venue":"G MSTI(ours) ResNet","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[3G References Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new mod- el and the kinetics dataset. In CVPR, pages 6299-6308, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Donahue et al., 2015] Jeffrey Donahue, Lisa Anne Hen- dricks, Sergio Guadarrama, Marcus Rohrbach, Subhashi- ni Venugopalan, Kate Saenko, and Trevor Darrell. Long- term recurrent convolutional networks for visual recogni- tion and description. In CVPR, pages 2625-2634, 2015.","title":"Longterm recurrent convolutional networks for visual recognition and description","authors":"Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feichtenhofer et al., 2016] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes. Spatiotemporal residual net- works for video action recognition. In NIPS, pages 3468- 3476, 2016.","title":"Spatiotemporal residual networks for video action recognition","authors":"Christoph Feichtenhofer, Axel Pinz, Richard Wildes","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Feichtenhofer et al., 2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 6202-6211, 2019.","title":"Slowfast networks for video recognition","authors":"Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2019] Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. arXiv preprint arXiv:1904.01169, 2019.","title":"Res2net: A new multi-scale backbone architecture","authors":"Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, Philip Torr","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2018] Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Liming Wang, and Shilei Wen. Stnet: Local and global spatial-temporal modeling for action recognition. arXiv preprint arXiv:1811.01549, 2018.","title":"Stnet: Local and global spatial-temporal modeling for action recognition","authors":"Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Liming Wang, Shilei Wen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In CVPR, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karpathy et al., 2014] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolu- tional neural networks. In CVPR, pages 1725-1732, 2014.","title":"Large-scale video classification with convolutional neural networks","authors":"Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, Li Fei-Fei","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kay et al., 2017] Will Kay, Joao Carreira, Karen Si- monyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Kay","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kuehne et al., 2013] Hilde Kuehne, Hueihan Jhuang, Rain- er Stiefelhagen, and Thomas Serre. Hmdb51: A large video database for human motion recognition. In HPCSE, pages 571-582. Springer, 2013.","title":"Thomas Serre. Hmdb51: A large video database for human motion recognition","authors":"Kuehne","year":"2013","venue":"HPCSE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2016] Jiawei Liu, Zheng-Jun Zha, QI Tian, Dong Liu, Ting Yao, Qiang Ling, and Tao Mei. Multi-scale triplet cnn for person re-identification. In ACM Multime- dia, pages 192-196, 2016.","title":"Multi-scale triplet cnn for person re-identification","authors":"Jiawei Liu, Zheng-Jun Zha, Dong Qi Tian, Ting Liu, Qiang Yao, Tao Ling, Mei","year":"2016","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann ma- chines. In ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qiu et al., 2017] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, pages 5533-5541, 2017.","title":"Learning spatio-temporal representation with pseudo-3d residual networks","authors":"Qiu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran et al., 2017] Du Tran, Jamie Ray, Zheng Shou, Shih- Fu Chang, and Manohar Paluri. Convnet architecture search for spatiotemporal feature learning. arXiv preprint arXiv:1708.05038, 2017.","title":"Convnet architecture search for spatiotemporal feature learning","authors":"Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, Manohar Paluri","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tran et al., 2018] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450-6459, 2018.","title":"A closer look at spatiotemporal convolutions for action recognition","authors":"Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann Lecun, Manohar Paluri","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pages 20- 36. Springer, 2016.","title":"Temporal segment networks: Towards good practices for deep action recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018a] Limin Wang, Wei Li, Wen Li, and Luc Van Gool. Appearance-and-relation networks for video classification. In CVPR, pages 1430-1439, 2018.","title":"Appearance-and-relation networks for video classification","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2018b] Xiaolong Wang, Ross Girshick, Abhi- nav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2019a] Haoze Wu, Jiawei Liu, Zheng-Jun Zha, Zhenzhong Chen, and Xiaoyan Sun. Mutually rein- forced spatio-temporal convolutional tube for human ac- tion recognition. In IJCAI, pages 968-974, 2019.","title":"Mutually reinforced spatio-temporal convolutional tube for human action recognition","authors":"Wu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/136"},{"position":29,"raw":"[Wu et al., 2019b] Haoze Wu, Zheng-Jun Zha, Xin Wen, Zhenzhong Chen, Dong Liu, and Xuejin Chen. Cross-fiber spatial-temporal co-enhanced networks for video action recognition. In ACM Multimedia, pages 620-628, 2019.","title":"Cross-fiber spatial-temporal co-enhanced networks for video action recognition","authors":"Wu","year":"2019","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xie et al., 2018] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. In ECCV, pages 305-321, 2018.","title":"Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification","authors":"Xie","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2018] Yizhou Zhou, Xiaoyan Sun, Zheng-Jun Zha, and Wenjun Zeng. Mict: Mixed 3d/2d convolutional tube for human action recognition. In CVPR, pages 449- 458, 2018.","title":"Mict: Mixed 3d/2d convolutional tube for human action recognition","authors":"Yizhou Zhou, Xiaoyan Sun, Zheng-Jun Zha, Wenjun Zeng","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}