{"type":"ScholarlyArticle","title":"Mutually Reinforced Spatio-Temporal Convolutional Tube for Human Action Recognition","authors":[{"name":"Haoze Wu","slug":"haoze-wu","given":"Haoze","family":"Wu","affiliation":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application , University of Science and Technology of China"},{"name":"Jiawei Liu","slug":"jiawei-liu","given":"Jiawei","family":"Liu","affiliation":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application , University of Science and Technology of China"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application , University of Science and Technology of China"},{"name":"Zhenzhong Chen","slug":"zhenzhong-chen","given":"Zhenzhong","family":"Chen","affiliation":"School of Remote Sensing and Information Engineering , Wuhan University"},{"name":"Xiaoyan Sun","slug":"xiaoyan-sun","given":"Xiaoyan","family":"Sun","affiliation":"Intelligent Multimedia Group , Microsoft Research Asia"}],"abstract":"Recent works use 3D convolutional neural networks to explore spatio-temporal information for human action recognition. However, they either ignore the correlation between spatial and temporal features or suffer from high computational cost by spatio-temporal features extraction. In this work, we propose a novel and efficient Mutually Reinforced Spatio-Temporal Convolutional Tube (MRST) for human action recognition. It decomposes 3D inputs into spatial and temporal representations, mutually enhances both of them by exploiting the interaction of spatial and temporal information and selectively emphasizes informative spatial appearance and temporal motion, meanwhile reducing the complexity of structure. Moreover, we design three types of MRSTs according to the different order of spatial and temporal information enhancement, each of which contains a spatio-temporal decomposition unit, a mutually reinforced unit and a spatio-temporal fusion unit. An end-to-end deep network, MRST-Net, is also proposed based on the MRSTs to better explore spatio-temporal information in human actions. Extensive experiments show MRST-Net yields the best performance, compared to state-of-the-art approaches.","doi":"10.24963/ijcai.2019/136","url":"https://ijcai.org/proceedings/2019/136","pdf":"https://ijcai.org/proceedings/2019/0136.pdf","markdown":"https://ijcai.org/proceedings/2019/136.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/136","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":136,"pages":{"first":968,"last":974},"keywords":["Computer Vision: 2D and 3D Computer Vision","Computer Vision: Action Recognition"],"videos":[],"references":[{"position":1,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new mod- el and the kinetics dataset. In CVPR, pages 6299-6308, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Yunpeng Chen, Yannis Kalantidis, Jian- shu Li, Shuicheng Yan, and Jiashi Feng. Multi-fiber net- works for video recognition. In ECCV, pages 352-367, 2018.","title":"Multi-fiber networks for video recognition","authors":"Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, Jiashi Feng","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Donahue et al., 2015] Jeffrey Donahue, Lisa Anne Hen- dricks, Sergio Guadarrama, Marcus Rohrbach, Subhashi- ni Venugopalan, Kate Saenko, and Trevor Darrell. Long- term recurrent convolutional networks for visual recogni- tion and description. In CVPR, pages 2625-2634, 2015.","title":"Longterm recurrent convolutional networks for visual recognition and description","authors":"Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feichtenhofer et al., 2017] Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for video action recognition. In CVPR, pages 4768-4777, 2017.","title":"Spatiotemporal multiplier networks for video action recognition","authors":"Christoph Feichtenhofer, Axel Pinz, Richard P Wildes","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2018] Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Liming Wang, and Shilei Wen. Stnet: Local and global spatial-temporal modeling for action recognition. arXiv preprint arXiv:1811.01549, 2018.","title":"Stnet: Local and global spatial-temporal modeling for action recognition","authors":"Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Liming Wang, Shilei Wen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory, volume 9. MIT Press, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In CVPR, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ji et al., 2013] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. TPAMI, 35(1):221-231, 2013.","title":"3d convolutional neural networks for human action recognition","authors":"Shuiwang Ji, Wei Xu, Ming Yang, Kai Yu","year":"2013","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karpathy et al., 2014] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolu- tional neural networks. In CVPR, pages 1725-1732, 2014.","title":"Large-scale video classification with convolutional neural networks","authors":"Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, Li Fei-Fei","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kay et al., 2017] Will Kay, Joao Carreira, Karen Si- monyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Kay","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kuehne et al., 2013] Hilde Kuehne, Hueihan Jhuang, Rain- er Stiefelhagen, and Thomas Serre. Hmdb51: A large video database for human motion recognition. In HPCSE, pages 571-582. Springer, 2013.","title":"Thomas Serre. Hmdb51: A large video database for human motion recognition","authors":"Kuehne","year":"2013","venue":"HPCSE","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Qiu et al., 2017] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, pages 5533-5541, 2017.","title":"Learning spatio-temporal representation with pseudo-3d residual networks","authors":"Qiu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sun et al., 2015] Lin Sun, Kui Jia, Dit-Yan Yeung, and Bertram E Shi. Human action recognition using factorized spatio-temporal convolutional networks. In ICCV, pages 4597-4605, 2015.","title":"Human action recognition using factorized spatio-temporal convolutional networks","authors":"Lin Sun, Kui Jia, Dit-Yan Yeung, Bertram E Shi","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tran and Cheong, 2017] An Tran and Loong-Fah Cheong. Two-stream flow-guided convolutional attention networks for action recognition. In ICCV, pages 3110-3119, 2017.","title":"An Tran and Loong-Fah Cheong. Two-stream flow-guided convolutional attention networks for action recognition","authors":"Cheong Tran","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tran et al., 2017] Du Tran, Jamie Ray, Zheng Shou, Shih- Fu Chang, and Manohar Paluri. Convnet architecture search for spatiotemporal feature learning. arXiv preprint arXiv:1708.05038, 2017.","title":"Convnet architecture search for spatiotemporal feature learning","authors":"Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, Manohar Paluri","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tran et al., 2018] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450-6459, 2018.","title":"A closer look at spatiotemporal convolutions for action recognition","authors":"Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann Lecun, Manohar Paluri","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pages 20- 36. Springer, 2016.","title":"Temporal segment networks: Towards good practices for deep action recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2018a] Limin Wang, Wei Li, Wen Li, and Luc Van Gool. Appearance-and-relation networks for video classification. In CVPR, pages 1430-1439, 2018.","title":"Appearance-and-relation networks for video classification","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018b] Xiaolong Wang, Ross Girshick, Abhi- nav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2018] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. In ECCV, pages 305-321, 2018.","title":"Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification","authors":"Xie","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2018] Yizhou Zhou, Xiaoyan Sun, Zheng-Jun Zha, and Wenjun Zeng. Mict: Mixed 3d/2d convolutional tube for human action recognition. In CVPR, pages 449- 458, 2018.","title":"Mict: Mixed 3d/2d convolutional tube for human action recognition","authors":"Yizhou Zhou, Xiaoyan Sun, Zheng-Jun Zha, Wenjun Zeng","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}