{"type":"ScholarlyArticle","title":"Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation","authors":[{"name":"Chao Li","slug":"chao-li","given":"Chao","family":"Li","affiliation":"Research Institute"},{"name":"Qiaoyong Zhong","slug":"qiaoyong-zhong","given":"Qiaoyong","family":"Zhong","affiliation":"Research Institute"},{"name":"Di Xie","slug":"di-xie","given":"Di","family":"Xie","affiliation":"Research Institute"},{"name":"Shiliang Pu","slug":"shiliang-pu","given":"Shiliang","family":"Pu","affiliation":"Research Institute"}],"abstract":"Skeleton-based human action recognition has recently drawn increasing attentions with the availability of large-scale skeleton datasets. The most crucial factors for this task lie in two aspects: the intra-frame representation for joint co-occurrences and the inter-frame representation for skeletons' temporal evolutions. In this paper we propose an end-to-end convolutional co-occurrence feature learning framework. The co-occurrence features are learned with a hierarchical methodology, in which different levels of contextual information are aggregated gradually. Firstly point-level information of each joint is encoded independently. Then they are assembled into semantic representation in both spatial and temporal domains. Specifically, we introduce a global spatial aggregation scheme, which is able to learn superior joint co-occurrence features over local aggregation. Besides, raw skeleton coordinates as well as their temporal difference are integrated with a two-stream paradigm. Experiments show that our approach consistently outperforms other state-of-the-arts on action recognition and detection benchmarks like NTU RGB+D, SBU Kinect Interaction and PKU-MMD.","doi":"10.24963/ijcai.2018/109","url":"https://ijcai.org/proceedings/2018/109","pdf":"https://ijcai.org/proceedings/2018/0109.pdf","markdown":"https://ijcai.org/proceedings/2018/109.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/109","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":109,"pages":{"first":786,"last":792},"keywords":["Computer Vision: 2D and 3D Computer Vision","Machine Learning: Deep Learning","Computer Vision: Computer Vision","Computer Vision: Action Recognition"],"videos":[],"references":[{"position":1,"raw":"[Du et al., 2016] Yong Du, Yun Fu, and Liang Wang. Skele- ton based action recognition with convolutional neural net- work. In Pattern Recognition, pages 579-583, 2016.","title":"Skeleton based action recognition with convolutional neural network","authors":"Yong Du, Yun Fu, Liang Wang","year":"2016","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Girshick et al., 2014] Ross B Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierar- chies for accurate object detection and semantic segmen- tation. CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Jeff Ross B Girshick, Trevor Donahue, Jitendra Darrell, Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hussein et al., 2013] Mohamed E. Hussein, Marwan Torki, Mohammad A. Gowayyed, and Motaz El-Saban. Human action recognition using a temporal hierarchy of covari- ance descriptors on 3d joint locations. In IJCAI, pages 639-44, 2013.","title":"Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations","authors":"Mohamed E Hussein, Marwan Torki, Mohammad A Gowayyed, Motaz El-Saban","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/363"},{"position":5,"raw":"[Ji et al., 2014] Yanli Ji, Guo Ye, and Hong Cheng. Inter- active body part contrast mining for human interaction recognition. In ICMEW, pages 1-6, 2014.","title":"Interactive body part contrast mining for human interaction recognition","authors":"Yanli Ji, Guo Ye, Hong Cheng","year":"2014","venue":"ICMEW","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jin and Choi, 2012] Sou Young Jin and Ho Jin Choi. Essen- tial body-joint and atomic action detection for human ac- tivity recognition using longest common subsequence al- gorithm. In ICCV, pages 148-159, 2012.","title":"Essential body-joint and atomic action detection for human activity recognition using longest common subsequence algorithm","authors":"Jin, Choi ; ] Sou, Young Jin, Jin Ho, Choi","year":"2012","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ke et al., 2017] Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, and Farid Boussaid. A New Representation of Skeleton Sequences for 3D Action Recognition. In CVPR, July 2017.","title":"A New Representation of Skeleton Sequences for 3D Action Recognition","authors":"Ke","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Lei Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016] Yanghao Li, Cuiling Lan, Junliang Xing, Wenjun Zeng, Chunfeng Yuan, and Jiaying Liu. On- line human action detection using joint classification- regression recurrent neural networks. ECCV, pages 203- 220, 2016.","title":"Online human action detection using joint classificationregression recurrent neural networks","authors":"Li","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017a] Bo Li, Huahui Chen, Yucheng Chen, Yuchao Dai, and Mingyi He. Skeleton boxes: Solving skeleton based action detection with a single deep convo- lutional neural network. In ICMEW, pages 613-616, July 2017.","title":"Skeleton boxes: Solving skeleton based action detection with a single deep convolutional neural network","authors":"Li","year":"2017","venue":"ICMEW","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017b] Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu. Skeleton-based action recognition with con- volutional neural networks. In ICMEW, pages 597-600, July 2017.","title":"Skeleton-based action recognition with convolutional neural networks","authors":"Li","year":"2017","venue":"ICMEW","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal lstm with trust gates for 3d human action recognition. In ECCV, pages 816-833, 2016.","title":"Spatio-temporal lstm with trust gates for 3d human action recognition","authors":"Jun Liu, Amir Shahroudy, Dong Xu, Gang Wang","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2017] Chunhui Liu, Yueyu Hu, Yanghao Li, Si- jie Song, and Jiaying Liu. PKU-MMD: A large scale benchmark for continuous multi-modal human action un- derstanding. ACM Multimedia workshop, 2017.","title":"PKU-MMD: A large scale benchmark for continuous multi-modal human action understanding","authors":"Chunhui Liu, Yueyu Hu, Yanghao Li, Sijie Song, Jiaying Liu","year":"2017","venue":"ACM Multimedia workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sainath et al., 2015] T. N. Sainath, O. Vinyals, A. Senior, and H. Sak. Convolutional, long short-term memory, fully connected deep neural networks. In 2015 IEEE Interna- tional Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 4580-4584, April 2015.","title":"Convolutional, long short-term memory, fully connected deep neural networks","authors":"T N Sainath, O Vinyals, A Senior, H Sak","year":"2015","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shahroudy et al., 2016] Amir Shahroudy, Jun Liu, Tian- Tsong Ng, and Gang Wang. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. In CVPR, June 2016.","title":"NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis","authors":"Amir Shahroudy, Jun Liu, Tian-Tsong Ng, Gang Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Wein- berger, editors, NIPS, pages 568-576.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman ; Simonyan, Zisserman","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2017] Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu. An end-to-end spatio- temporal attention model for human action recognition from skeleton data. In AAAI, pages 4263-4270, 2017.","title":"An end-to-end spatiotemporal attention model for human action recognition from skeleton data","authors":"Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jiaying Liu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2014] Jiang Wang, Zicheng Liu, Ying Wu, and Junsong Yuan. Learning actionlet ensemble for 3d human action recognition. TPAMI, 36(5):914, 2014.","title":"Learning actionlet ensemble for 3d human action recognition","authors":"Jiang Wang, Zicheng Liu, Ying Wu, Junsong Yuan","year":"2014","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xia et al., 2012] L. Xia, C. C. Chen, and J. K. Aggarwal. View invariant human action recognition using histograms of 3d joints. In CVPRW, pages 20-27, June 2012.","title":"View invariant human action recognition using histograms of 3d joints","authors":"L Xia, C C Chen, J K Aggarwal","year":"2012","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2017] Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. In ICCV, Oct 2017.","title":"R-C3D: Region Convolutional 3D Network for Temporal Activity Detection","authors":"Xu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yun et al., 2012] Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L. Berg, and Dimitris Samaras. Two-person interaction detection using body-pose features and multiple instance learning. In CVPRW. IEEE, 2012.","title":"Two-person interaction detection using body-pose features and multiple instance learning","authors":"Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L Berg, Dimitris Samaras","year":"2012","venue":"CVPRW. IEEE","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2017] Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, and Nanning Zheng. View adaptive recurrent neural networks for high perfor- mance human action recognition from skeleton data. In ICCV, Oct 2017.","title":"View adaptive recurrent neural networks for high performance human action recognition from skeleton data","authors":"Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, Nanning Zheng","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhong et al., 2017] Qiaoyong Zhong, Chao Li, Yingying Zhang, Di Xie, Shicai Yang, and Shiliang Pu. Cascade region proposal and global context for deep object detec- tion. arXiv preprint arXiv:1710.10749, 2017.","title":"Cascade region proposal and global context for deep object detection","authors":"Qiaoyong Zhong, Chao Li, Yingying Zhang, Di Xie, Shicai Yang, Shiliang Pu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2016] Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, and Xiaohui Xie. Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. In AAAI, pages 3697-3703, 2016.","title":"Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks","authors":"Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, Xiaohui Xie","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}