{"type":"ScholarlyArticle","title":"Memory Attention Networks for Skeleton-based Action Recognition","authors":[{"name":"Chunyu Xie","slug":"chunyu-xie","given":"Chunyu","family":"Xie","affiliation":"School of Automation Science and Electrical Engineering , Beihang University , Beijing , China"},{"name":"Ce Li","slug":"ce-li","given":"Ce","family":"Li","affiliation":"China University of Mining & Technology , Beijing , China"},{"name":"Baochang Zhang","slug":"baochang-zhang","given":"Baochang","family":"Zhang","affiliation":"School of Automation Science and Electrical Engineering , Beihang University , Beijing , China"},{"name":"Chen Chen","slug":"chen-chen","given":"Chen","family":"Chen","affiliation":"Department of Electrical and Computer Engineering , University of North Carolina at Charlotte"},{"name":"Jungong Han","slug":"jungong-han","given":"Jungong","family":"Han","affiliation":"School of Computing & Communications , Lancaster University , LA1 4YW , UK"},{"name":"Jianzhuang Liu","slug":"jianzhuang-liu","given":"Jianzhuang","family":"Liu","affiliation":"Noah's Ark Lab , Huawei Technologies Co. Ltd. , China"}],"abstract":"Skeleton-based action recognition task is entangled with complex spatio-temporal variations of skeleton joints, and remains challenging for Recurrent Neural Networks (RNNs). In this work, we propose a temporal-then-spatial recalibration scheme to alleviate such complex variations, resulting in an end-to-end Memory Attention Networks (MANs) which consist of a Temporal Attention Recalibration Module (TARM) and a Spatio-Temporal Convolution Module (STCM). Specifically, the TARM is deployed in a residual learning module that employs a novel attention learning network to recalibrate the temporal attention of frames in a skeleton sequence. The STCM treats the attention calibrated skeleton joint sequences as images and leverages the Convolution Neural Networks (CNNs) to further model the spatial and temporal information of skeleton data. These two modules (TARM and STCM) seamlessly form a single network architecture that can be trained in an end-to-end fashion. MANs significantly boost the performance of skeleton-based action recognition and achieve the best results on four challenging benchmark datasets: NTU RGB+D, HDM05, SYSU-3D and UT-Kinect.","doi":"10.24963/ijcai.2018/227","url":"https://ijcai.org/proceedings/2018/227","pdf":"https://ijcai.org/proceedings/2018/0227.pdf","markdown":"https://ijcai.org/proceedings/2018/227.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/227","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Humans and AI","track":"Main track","number":227,"pages":{"first":1639,"last":1645},"keywords":["Computer Vision: Action Recognition","Machine Learning: Deep Learning","Humans and AI: Human-Computer Interaction","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[],"references":[{"position":1,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merriënboer, C ¸aglar Gülc ¸ehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statisti- cal machine translation. In Empirical Methods in Natural Language Processing, pages 1724-1734, 2014.","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ding and Fan, 2016] Meng Ding and Guoliang Fan. Artic- ulated and generalized gaussian kernel correlation for hu- man pose estimation. IEEE Transactions on Image Pro- cessing A Publication of the IEEE Signal Processing Soci- ety, 25(2):776, 2016.","title":"Articulated and generalized gaussian kernel correlation for human pose estimation","authors":"Meng Ding, Guoliang Fan","year":"2016","venue":"IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Du et al., 2015] Yong Du, Wei Wang, and Liang Wang. Hi- erarchical recurrent neural network for skeleton based ac- tion recognition. In CVPR, pages 1110-1118, 2015.","title":"Hierarchical recurrent neural network for skeleton based action recognition","authors":"Yong Du, Wei Wang, Liang Wang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":"http://www.deeplearningbook.org","internal_url":null},{"position":5,"raw":"[Han et al., 2017] Fei Han, Brain Reily, William Hoff, and Hao Zhang. Space-time rep-resentation of people based on 3D skeletal data: A review. Computer Vision and Image Understanding, 158:85-105, 2017.","title":"Space-time rep-resentation of people based on 3D skeletal data: A review","authors":"Fei Han, Brain Reily, William Hoff, Hao Zhang","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2015] Jianfang Hu, Weishi Zheng, Jianhuang Lai, and Jianguo Zhang. Jointly learning heterogeneous fea- tures for RGB-D activity recognition. In CVPR, pages 5344-5352, 2015. 4, 6","title":"Jointly learning heterogeneous features for RGB-D activity recognition","authors":"Hu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. In CVPR, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2015] Min Jiang, Jun Kong, George Bebis, and Hongtao Huo. Informative joints based human action recognition using skeleton contexts. Signal Processing: Image Communication, 33:29-40, 2015.","title":"Informative joints based human action recognition using skeleton contexts","authors":"Jiang","year":"2015","venue":"Signal Processing: Image Communication","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johansson, 1973] Gunnar Johansson. Visual perception of biological motion and a model for its analysis. Perception and Psychophysic, 14(2):201-211, 1973.","title":"Visual perception of biological motion and a model for its analysis","authors":"Gunnar Johansson, Johansson","year":"1973","venue":"Perception and Psychophysic","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ke et al., 2017] Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, and Farid Boussaid. A new representation of skeleton sequences for 3D action recog- nition. In CVPR, 2017. 1, 2, 6","title":"A new representation of skeleton sequences for 3D action recognition","authors":"Ke","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lea et al., 2016] Colin Lea, Reiter Austin Vidal, René, and Gregory D. Hager. Temporal convolutional networks: A unified approach to action segmentation. ECCV, pages 47- 54, 2016.","title":"Temporal convolutional networks: A unified approach to action segmentation","authors":"Colin Lea, Austin Reiter, René Vidal, Gregory D Hager","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017] Wenbo Li, Longyin Wen, Ming-Ching Chang, Ser Nam Lim, and Siwei Lyu. Adaptive RNN tree for large-scale human action recognition. In ICCV, pages 1444-1452, 2017. 1, 6","title":"Adaptive RNN tree for large-scale human action recognition","authors":"Wenbo Li, Longyin Wen, Ming-Ching Chang, Ser, Nam Lim, Siwei Lyu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2016] Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal LSTM with trust gates for 3D human action recognition. In ECCV, pages 816-833, 2016. 1, 4, 6","title":"Spatio-temporal LSTM with trust gates for 3D human action recognition","authors":"Jun Liu, Amir Shahroudy, Dong Xu, Gang Wang","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2017a] Jun Liu, Gang Wang, Ping Hu, Ling-Yu Duan, and Alex C. Kot. Global context-aware attention LSTM network for 3D action recognition. In CVPR, pages 1647-1656, 2017. 1, 4, 6","title":"Global context-aware attention LSTM network for 3D action recognition","authors":"Liu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2017b] Mengyuan Liu, Hong Liu, and Chen Chen. Enhanced skeleton visualization for view invariant human action recognition. PR, 68:346-362, 2017. 1, 2","title":"Enhanced skeleton visualization for view invariant human action recognition","authors":"Liu","year":"2017","venue":"PR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Müller et al., 2005] M. Müller, T. Röder, and M. Clausen. Efficient content-based retrieval of motion capture data. ACM Transactions on Graphic, 24(3):677-685, 2005.","title":"Efficient content-based retrieval of motion capture data","authors":"M Müller, T Röder, M Clausen","year":"2005","venue":"ACM Transactions on Graphic","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Presti and La Cascia, 2016] Liliana Lo Presti and Marco La Cascia. 3D skeleton-based human action classification. PR, 53:130-147, 2016.","title":"Liliana Lo Presti and Marco La Cascia. 3D skeleton-based human action classification","authors":"La Presti, Cascia","year":"2016","venue":"PR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Raptis et al., 2011] Michalis Raptis, Darko Kirovski, and Hugues Hoppe. Real-time classification of dance ges- tures from skeleton animation. In Proceedings of the 2011 ACM SIGGRAPH/Eurographics symposium on computer animation, pages 147-156. ACM, 2011.","title":"Real-time classification of dance gestures from skeleton animation","authors":"Michalis Raptis, Darko Kirovski, Hugues Hoppe","year":"2011","venue":"Proceedings of the 2011 ACM SIGGRAPH/Eurographics symposium on computer animation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shahroudy et al., 2016] Amir Shahroudy, Jun Liu, Tian- Tsong Ng, and Gang Wang. NTU RGB+D: A large scale dataset for 3D human activity analysis. In CVPR, pages 1010-1019, 2016.","title":"NTU RGB+D: A large scale dataset for 3D human activity analysis","authors":"Amir Shahroudy, Jun Liu, Tian-Tsong Ng, Gang Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2017] Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu. An end-to-end spatio- temporal attention model for human action recognition from skeleton data. In AAAI, pages 4263-4270, 2017. 1, 2, 4, 5, 6","title":"An end-to-end spatiotemporal attention model for human action recognition from skeleton data","authors":"Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jiaying Liu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Wang, 2017] Hongsong Wang and Liang Wang. Modeling temporal dynamics and spatial configuration of actions using two-stream recurrent neural networks. In CVPR, 2017.","title":"Modeling temporal dynamics and spatial configuration of actions using two-stream recurrent neural networks","authors":"Wang, Wang ;, Hongsong Wang, Liang Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xia et al., 2012] Lu Xia, Chiachih Chen, and Jake K Ag- garwal. View invariant human action recognition using histograms of 3D joints. In CVPRW, pages 20-27, 2012. 1, 4","title":"View invariant human action recognition using histograms of 3D joints","authors":"Lu Xia, Chiachih Chen, Jake K Aggarwal","year":"2012","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2016] Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and HanLi Wang. Real-time action recogni- tion with enhanced motion vector CNNs. In CVPR, pages 2718-2726, 2016.","title":"Real-time action recognition with enhanced motion vector CNNs","authors":"Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, Hanli Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2017] Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, and Nanning Zheng. View adaptive recurrent neural networks for high perfor- mance human action recognition from skeleton data. In ICCV, 2017.","title":"View adaptive recurrent neural networks for high performance human action recognition from skeleton data","authors":"Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, Nanning Zheng","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2016] Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, and Xiaohui Xie. Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. In AAAI, pages 3697-3704, 2016. 1, 5, 6","title":"Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks","authors":"Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, Xiaohui Xie","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}