{"type":"ScholarlyArticle","title":"WALKING WALKing walking: Action Recognition from Action Echoes","authors":[{"name":"Qianli Ma","slug":"qianli-ma","given":"Qianli","family":"Ma","affiliation":"† School of Computer Science and Engineering , South China University of Technology , Guangzhou , China"},{"name":"Lifeng Shen","slug":"lifeng-shen","given":"Lifeng","family":"Shen","affiliation":"† School of Computer Science and Engineering , South China University of Technology , Guangzhou , China"},{"name":"Enhuan Chen","slug":"enhuan-chen","given":"Enhuan","family":"Chen","affiliation":"† School of Computer Science and Engineering , South China University of Technology , Guangzhou , China"},{"name":"Shuai Tian","slug":"shuai-tian","given":"Shuai","family":"Tian","affiliation":"† School of Computer Science and Engineering , South China University of Technology , Guangzhou , China"},{"name":"Jiabing Wang","slug":"jiabing-wang","given":"Jiabing","family":"Wang","affiliation":"† School of Computer Science and Engineering , South China University of Technology , Guangzhou , China"},{"name":"Garrison W. Cottrell","slug":"garrison-w-cottrell","given":"Garrison W.","family":"Cottrell","affiliation":"‡ Department of Computer Science and Engineering , University of California , San Diego , CA , USA"}],"abstract":"Recognizing human actions represented by 3D trajectories of skeleton joints is a challenging machine learning task. In this paper, the 3D skeleton sequences are regarded as multivariate time series, and their dynamics and multiscale features are efficiently learned from action echo states. Specifically, first the skeleton data from the limbs and trunk are projected into five high dimensional nonlinear spaces, that are randomly generated by five dynamic, training-free recurrent networks, i.e., the reservoirs of echo state networks (ESNs). In this way, the history of the time series is represented as nonlinear echo states of actions. We then use a single multiscale convolutional layer to extract multiscale features from the echo states, and maintain multiscale temporal invariance by a max-over-time pooling layer. We propose two multi-step fusion strategies to integrate the spatial information over the five parts of the human physical structure. Finally, we learn the label distribution using softmax. With one training-free recurrent layer and only layer of convolution, our Convolutional Echo State Network (ConvESN) is a very efficient end-to-end model, and achieves state-of-the-art performance on four skeleton benchmark data sets.","doi":"10.24963/ijcai.2017/342","url":"https://ijcai.org/proceedings/2017/342","pdf":"https://ijcai.org/proceedings/2017/0342.pdf","markdown":"https://ijcai.org/proceedings/2017/342.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/342","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":342,"pages":{"first":2457,"last":2463},"keywords":["Robotics and Vision: Vision and Perception","Machine Learning: Neural Networks","Machine Learning: Time-series/Data Streams"],"videos":[],"references":[{"position":1,"raw":"[Chen and Liu, 2016] Chen Chen and Mengyuan Liu. 3d ac- tion recognition using multi-temporal depth motion maps and fisher vector. In International Joint Conference on Ar- tificial Intelligence, pages 3331-3337, July 2016.","title":"3d action recognition using multi-temporal depth motion maps and fisher vector","authors":"Chen, Liu, Chen Chen, Mengyuan Liu","year":"2016","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/471"},{"position":2,"raw":"[Cho and Chen, 2014] Kyunghyun Cho and Xi Chen. Classi- fying and visualizing motion capture sequences using deep neural networks. In International Conference on Comput- er Vision Theory and Applications, pages 122-130, 2014.","title":"Classifying and visualizing motion capture sequences using deep neural networks","authors":"Chen ; Cho, Kyunghyun Cho, Xi Chen","year":"2014","venue":"International Conference on Computer Vision Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493-2537, November 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devanne et al., 2013] Maxime Devanne, Hazem Wannous, Stefano Berretti, Pietro Pala, Mohamed Daoudi, and Al- berto Del Bimbo. Space-time pose representation for 3d human action recognition. In International Conference on Image Analysis and Processing, pages 456-464. Springer, 2013.","title":"Space-time pose representation for 3d human action recognition","authors":"Maxime Devanne, Hazem Wannous, Stefano Berretti, Pietro Pala, Mohamed Daoudi, Alberto Del Bimbo","year":"2013","venue":"International Conference on Image Analysis and Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2015] Yong Du, Wei Wang, and Liang Wang. Hi- erarchical recurrent neural network for skeleton based ac- tion recognition. In 2015 IEEE Conference on Computer Vision and Pattern Recognition, pages 1110-1118, June 2015.","title":"Hierarchical recurrent neural network for skeleton based action recognition","authors":"Yong Du, Wei Wang, Liang Wang","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gong et al., 2014] Dian Gong, Gerard Medioni, and Xue- mei Zhao. Structured time series analysis for human action segmentation and recognition. IEEE Trans. Pattern Anal. Mach. Intell., 36(7):1414-1427, July 2014.","title":"Structured time series analysis for human action segmentation and recognition","authors":"Dian Gong, Gerard Medioni, Xuemei Zhao","year":"2014","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gowayyed et al., 2013] Mohammad A. Gowayyed, Mar- wan Torki, Mohamed E. Hussein, and Motaz El-Saban. Histogram of oriented displacements (hod): Describing trajectories of human joints for action recognition. In International Joint Conference on Artificial Intelligence, pages 1351-1357, 2013.","title":"Histogram of oriented displacements (hod): Describing trajectories of human joints for action recognition","authors":"Mohammad A Gowayyed, Marwan Torki, Mohamed E Hussein, Motaz El-Saban","year":"2013","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/203"},{"position":8,"raw":"[Hussein et al., 2013] Mohamed E Hussein, Marwan Torki, Mohammad Abdelaziz Gowayyed, and Motaz El-Saban. Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations. In Inter- national Joint Conference on Artificial Intelligence, vol- ume 13, pages 2466-2472, 2013.","title":"Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations","authors":"Mohamed E Hussein, Marwan Torki, Mohammad Abdelaziz Gowayyed, Motaz El-Saban","year":"2013","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/363"},{"position":9,"raw":"[Jaeger and Haas, 2004] Herbert Jaeger and Harald Haas. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science, 304(5667):78-80, 2004.","title":"Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication","authors":"Haas Jaeger, Herbert Jaeger, Harald Haas","year":"2004","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kerola et al., 2014] Tommi Kerola, Nakamasa Inoue, and Koichi Shinoda. Spectral graph skeletons for 3d action recognition. In Asian Conference on Computer Vision, pages 417-432. Springer, 2014.","title":"Spectral graph skeletons for 3d action recognition","authors":"Tommi Kerola, Nakamasa Inoue, Koichi Shinoda","year":"2014","venue":"Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arX- iv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2010] Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based on a bag of 3d points. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 9-14, June 2010.","title":"Action recognition based on a bag of 3d points","authors":"Wanqing Li, Zhengyou Zhang, Zicheng Liu","year":"2010","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lo Presti et al., 2015] Liliana Lo Presti, Marco La Cascia, Stan Sclaroff, and Octavia Camps. Hankelet-based dy- namical systems modeling for 3d action recognition. Im- age Vision Comput., 44(C):29-43, December 2015.","title":"Hankelet-based dynamical systems modeling for 3d action recognition","authors":"Liliana Lo Presti, Marco, La Cascia, Stan Sclaroff, Octavia Camps","year":"2015","venue":"Image Vision Comput","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lukoševičius and Jaeger, 2009] Mantas Lukoševičius and Herbert Jaeger. Survey: Reservoir computing approach- es to recurrent neural network training. Comput. Sci. Rev., 3(3):127-149, August 2009.","title":"Mantas Lukoševičius and Herbert Jaeger. Survey: Reservoir computing approaches to recurrent neural network training","authors":"Jaeger Lukoševičius","year":"2009","venue":"Comput. Sci. Rev","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Müller et al., 2007] Meinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Björn Krüger, and Andreas Weber. Documentation mocap database hdm05. Technical Report CG-2007-2, Universität Bonn, June 2007.","title":"Documentation mocap database hdm05","authors":"Müller","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Presti and Cascia, 2016] Liliana Lo Presti and Marco La Cascia. 3d skeleton-based human action classification: A survey. Pattern Recognition, 53:130 -147, 2016.","title":"Liliana Lo Presti and Marco La Cascia. 3d skeleton-based human action classification: A survey","authors":"Cascia Presti","year":"2016","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Seidenari et al., 2013] Lorenzo Seidenari, Vincenzo Vara- no, Stefano Berretti, Alberto Del Bimbo, and Pietro Pala. Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses. In The IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 479-485, June 2013.","title":"Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses","authors":"Lorenzo Seidenari, Vincenzo Varano, Stefano Berretti, Alberto Del Bimbo, Pietro Pala","year":"2013","venue":"The IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shotton et al., 2013] Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. Real-time human pose recognition in parts from single depth images. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1297-1304, 2013.","title":"Real-time human pose recognition in parts from single depth images","authors":"Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, Andrew Blake","year":"2013","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Slama et al., 2015] Rim Slama, Hazem Wannous, Mo- hamed Daoudi, and Anuj Srivastava. Accurate 3d ac- tion recognition using learning on the grassmann mani- fold. Pattern Recogn., 48(2):556-567, February 2015.","title":"Accurate 3d action recognition using learning on the grassmann manifold","authors":"Slama","year":"2015","venue":"Pattern Recogn","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2016] Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu. An end-to-end spatio- temporal attention model for human action recognition from skeleton data. arXiv:1611.06067, 2016.","title":"An end-to-end spatiotemporal attention model for human action recognition from skeleton data","authors":"Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jiaying Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Steinier et al., 1972] Jean. Steinier, Yves. Termonia, Jules. Deltour, and Anal. Chem. Smoothing and differentiation of data by simplified least square procedure. Analytical Chemistry, 44(11):1906-9, 1972.","title":"Smoothing and differentiation of data by simplified least square procedure","authors":"Jean Steinier, Yves Termonia, Jules Deltour, Anal Chem","year":"1972","venue":"Analytical Chemistry","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vemulapalli et al., 2014] Raviteja Vemulapalli, Felipe Ar- rate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. In IEEE Conference on Computer Vision and Pattern Recognition, pages 588-595, 2014.","title":"Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group","authors":"Vemulapalli","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu and Shao, 2014] Di Wu and Ling Shao. Leveraging hi- erarchical parametric networks for skeletal joints based ac- tion segmentation and recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 724-731, 2014.","title":"Leveraging hierarchical parametric networks for skeletal joints based action segmentation and recognition","authors":"Di Wu, Ling Shao","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xia et al., 2012] Lu Xia, Chia Chih Chen, and J. K. Aggar- wal. View invariant human action recognition using his- tograms of 3d joints. In IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition Work- shops, pages 20-27, June 2012.","title":"View invariant human action recognition using histograms of 3d joints","authors":"Lu Xia, Chia Chih Chen, J K Aggarwal","year":"2012","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2016] Xikang Zhang, Yin Wang, Mengran Gou, Mario Sznaier, and Octavia Camps. Efficient tempo- ral sequence comparison and classification using gram ma- trix embeddings on a riemannian manifold. In The IEEE Conference on Computer Vision and Pattern Recognition, pages 4498-4507, June 2016.","title":"Efficient temporal sequence comparison and classification using gram matrix embeddings on a riemannian manifold","authors":"Xikang Zhang, Yin Wang, Mengran Gou, Mario Sznaier, Octavia Camps","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2013] Yu Zhu, Wenbin Chen, and Guodong Guo. Fusing spatiotemporal features and joints for 3d action recognition. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 486-491, June 2013.","title":"Fusing spatiotemporal features and joints for 3d action recognition","authors":"Yu Zhu, Wenbin Chen, Guodong Guo","year":"2013","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2016] Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, and Xiaohui Xie. Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. In The AAAI Conference on Artificial Intelligence, pages 3697- 3703, 2016.","title":"Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks","authors":"Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, Xiaohui Xie","year":"2016","venue":"The AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}