{"type":"ScholarlyArticle","title":"Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamics","authors":[{"name":"Yongyi Tang","slug":"yongyi-tang","given":"Yongyi","family":"Tang","affiliation":"School of Electronics and Information Technology , Sun Yat-sen University"},{"name":"Lin Ma","slug":"lin-ma","given":"Lin","family":"Ma","affiliation":"Tencent AI Lab"},{"name":"Wei Liu","slug":"wei-liu","given":"Wei","family":"Liu","affiliation":"Tencent AI Lab"},{"name":"Wei-Shi Zheng","slug":"wei-shi-zheng","given":"Wei-Shi","family":"Zheng","affiliation":"School of Data and Computer Science , Sun Yat-sen University"}],"abstract":"Human motion prediction aims at generating future frames of human motion based on an observed sequence of skeletons. Recent methods employ the latest hidden states of a recurrent neural network (RNN) to encode the historical skeletons, which can only address short-term prediction. In this work, we propose a motion context modeling by summarizing the historical human motion with respect to the current prediction. A modified highway unit (MHU) is proposed for efficiently eliminating motionless joints and estimating next pose given the motion context. Furthermore, we enhance the motion dynamic by minimizing the gram matrix loss for long-term motion prediction. Experimental results show that the proposed model can promisingly forecast the human future movements, which yields superior performances over related state-of-the-art approaches.\r\nMoreover, specifying the motion context with the activity labels enables our model to perform human motion transfer.","doi":"10.24963/ijcai.2018/130","url":"https://ijcai.org/proceedings/2018/130","pdf":"https://ijcai.org/proceedings/2018/0130.pdf","markdown":"https://ijcai.org/proceedings/2018/130.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/130","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":130,"pages":{"first":935,"last":941},"keywords":["Computer Vision: Motion and Tracking","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[],"references":[{"position":1,"raw":"[Aggarwal and Cai, 1997] Jake K Aggarwal and Quin Cai. Human motion analysis: A review. In Nonrigid and Articulated Mo- tion Workshop, 1997. Proceedings., IEEE, pages 90-102. IEEE, 1997.","title":"Human motion analysis: A review","authors":"Cai ; ] Aggarwal, K Jake, Quin Aggarwal, Cai","year":"1997","venue":"Nonrigid and Articulated Motion Workshop","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brand and Hertzmann, 2000] Matthew Brand and Aaron Hertz- mann. Style machines. In Proceedings of the 27th annual con- ference on Computer graphics and interactive techniques, pages 183-192. ACM Press/Addison-Wesley Publishing Co., 2000.","title":"Matthew Brand and Aaron Hertzmann. Style machines","authors":"Hertzmann Brand","year":"2000","venue":"Proceedings of the 27th annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2015] Yong Du, Wei Wang, and Liang Wang. Hierarchi- cal recurrent neural network for skeleton based action recogni- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1110-1118, 2015.","title":"Hierarchical recurrent neural network for skeleton based action recognition","authors":"Yong Du, Wei Wang, Liang Wang","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fragkiadaki et al., 2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346-4354, 2015.","title":"Recurrent network models for human dynamics","authors":"Katerina Fragkiadaki, Sergey Levine, Panna Felsen, Jitendra Malik","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghosh et al., 2017] Partha Ghosh, Jie Song, Emre Aksan, and Ot- mar Hilliges. Learning human motion models for long-term pre- dictions. arXiv preprint arXiv:1704.02827, 2017.","title":"Learning human motion models for long-term predictions","authors":"Partha Ghosh, Jie Song, Emre Aksan, Otmar Hilliges","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grassia, 1998] F Sebastian Grassia. Practical parameterization of rotations using the exponential map. Journal of graphics tools, 3(3):29-48, 1998.","title":"Practical parameterization of rotations using the exponential map","authors":"Grassia Sebastian","year":"1998","venue":"Journal of graphics tools","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2015] Jian-Fang Hu, Wei-Shi Zheng, Jianhuang Lai, and Jianguo Zhang. Jointly learning heterogeneous features for rgb- d activity recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5344-5352, 2015.","title":"Jointly learning heterogeneous features for rgbd activity recognition","authors":"Jian-Fang Hu, Wei-Shi Zheng, Jianhuang Lai, Jianguo Zhang","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2017] Zhiwu Huang, Chengde Wan, Thomas Probst, and Luc Van Gool. Deep learning on lie groups for skeleton- based action recognition. In Proceedings of the 2017 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 6099-6108. IEEE computer Society, 2017.","title":"Deep learning on lie groups for skeletonbased action recognition","authors":"Zhiwu Huang, Chengde Wan, Thomas Probst, Luc Van Gool","year":"2017","venue":"Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ionescu et al., 2014] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environ- ments. IEEE Transactions on Pattern Analysis and Machine In- telligence, 36(7):1325-1339, jul 2014.","title":"Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments","authors":"Catalin Ionescu, Dragos Papava, Vlad Olaru, Cristian Sminchisescu","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jain et al., 2016] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio- temporal graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5308-5317, 2016.","title":"Structural-rnn: Deep learning on spatiotemporal graphs","authors":"Ashesh Jain, Silvio Amir R Zamir, Ashutosh Savarese, Saxena","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Chan, 2014] Sijin Li and Antoni B Chan. 3d human pose estimation from monocular images with deep convolutional neu- ral network. In Asian Conference on Computer Vision, pages 332-347. Springer, 2014.","title":"3d human pose estimation from monocular images with deep convolutional neural network","authors":"Chan ; ] Sijin Li, Antoni B Li, Chan","year":"2014","venue":"Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2016] Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal lstm with trust gates for 3d human action recognition. In European Conference on Computer Vision, pages 816-833. Springer, 2016.","title":"Spatio-temporal lstm with trust gates for 3d human action recognition","authors":"Jun Liu, Amir Shahroudy, Dong Xu, Gang Wang","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Martinez et al., 2017] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neu- ral networks. arXiv preprint arXiv:1705.02445, 2017.","title":"On human motion prediction using recurrent neural networks","authors":"Julieta Martinez, Michael J Black, Javier Romero","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104- 3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Taylor et al., 2007] Graham W Taylor, Geoffrey E Hinton, and Sam T Roweis. Modeling human motion using binary latent vari- ables. In Advances in neural information processing systems, pages 1345-1352, 2007.","title":"Modeling human motion using binary latent variables","authors":"Geoffrey E Graham W Taylor, Sam T Hinton, Roweis","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tekin et al., 2017] Bugra Tekin, Pablo Marquez Neila, Mathieu Salzmann, and Pascal Fua. Learning to fuse 2d and 3d image cues for monocular body pose estimation. In International Confer- ence on Computer Vision (ICCV), number EPFL-CONF-230311, 2017.","title":"Learning to fuse 2d and 3d image cues for monocular body pose estimation","authors":"Pablo Bugra Tekin, Mathieu Marquez Neila, Pascal Salzmann, Fua","year":"2017","venue":"International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vemulapalli et al., 2014] Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 588-595, 2014.","title":"Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group","authors":"Vemulapalli","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2014] Jiang Wang, Zicheng Liu, and Ying Wu. Learn- ing actionlet ensemble for 3d human action recognition. In Human Action Recognition with Depth Cameras, pages 11-40. Springer, 2014.","title":"Learning actionlet ensemble for 3d human action recognition","authors":"Jiang Wang, Zicheng Liu, Ying Wu","year":"2014","venue":"Human Action Recognition with Depth Cameras","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yasin et al., 2016] Hashim Yasin, Umar Iqbal, Bjorn Kruger, An- dreas Weber, and Juergen Gall. A dual-source approach for 3d pose estimation from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4948-4956, 2016.","title":"A dual-source approach for 3d pose estimation from a single image","authors":"Hashim Yasin, Umar Iqbal, Bjorn Kruger, Andreas Weber, Juergen Gall","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zilly et al., 2016] Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway net- works. arXiv preprint arXiv:1607.03474, 2016.","title":"Recurrent highway networks","authors":"Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, Jürgen Schmidhuber","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zimo et al., 2017] Li Zimo, Yi Xiao, He Chong, and Li Hao. Auto- conditioned lstm network for extended complex human motion synthesis. arXiv preprint arXiv:1707.05363, 2017.","title":"Autoconditioned lstm network for extended complex human motion synthesis","authors":"Li Zimo, Yi Xiao, He Chong, Li Hao","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}