{"type":"ScholarlyArticle","title":"Few-shot Human Motion Prediction via Learning Novel Motion Dynamics","authors":[{"name":"Chuanqi Zang","slug":"chuanqi-zang","given":"Chuanqi","family":"Zang","affiliation":"Beijing Laboratory of Intelligent Information Technology , Beijing Institute of Technology , China; Golisano College of Computing and Information Sciences , Rochester Institute of Technology , USA"},{"name":"Mingtao Pei","slug":"mingtao-pei","given":"Mingtao","family":"Pei","affiliation":"Beijing Laboratory of Intelligent Information Technology , Beijing Institute of Technology , China"},{"name":"Yu Kong","slug":"yu-kong","given":"Yu","family":"Kong","affiliation":"Golisano College of Computing and Information Sciences , Rochester Institute of Technology , USA"}],"abstract":"Human motion prediction is a task where we anticipate future motion based on past observation. Previous approaches rely on the access to large datasets of skeleton data, and thus are difficult to be generalized to novel motion dynamics with limited training data. In our work, we propose a novel approach named Motion Prediction Network (MoPredNet) for few-short human motion prediction. MoPredNet can be adapted to predicting new motion dynamics using limited data, and it elegantly captures long-term dependency in motion dynamics. Specifically, MoPredNet dynamically selects the most informative poses in the streaming motion data as masked poses. In addition, MoPredNet improves its encoding capability of motion dynamics by adaptively learning spatio-temporal structure from the observed poses and masked poses. We also propose to adapt MoPredNet to novel motion dynamics based on accumulated motion experiences and limited novel motion dynamics data. Experimental results show that our method achieves better performance over state-of-the-art methods in motion prediction.","doi":"10.24963/ijcai.2020/118","url":"https://ijcai.org/proceedings/2020/118","pdf":"https://ijcai.org/proceedings/2020/0118.pdf","markdown":"https://ijcai.org/proceedings/2020/118.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/118","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":118,"pages":{"first":846,"last":852},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Motion and Tracking"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24648/file_24648.m3u8","length_sec":301,"provider":"hls"}],"references":[{"position":1,"raw":"[Bai et al., 2018] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.","title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling","authors":"Bai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Behl et al., 2017] Aseem Behl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger. Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation in autonomous driving scenarios? In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.","title":"Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation in autonomous driving scenarios","authors":"Aseem Behl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, Andreas Geiger","year":"2017","venue":"The IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. In Ad- vances in Neural Information Processing Systems, pages 1171-1179, 2015.","title":"Scheduled sampling for sequence prediction with recurrent neural networks","authors":"Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126- 1135. JMLR. org, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fragkiadaki et al., 2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346-4354, 2015.","title":"Recurrent network models for human dynamics","authors":"Katerina Fragkiadaki, Sergey Levine, Panna Felsen, Jitendra Malik","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gidaris and Komodakis, 2018] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4367- 4375, 2018.","title":"Dynamic few-shot visual learning without forgetting","authors":"Spyros Gidaris, Nikos Komodakis","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gopalakrishnan et al., 2019] Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia. A neural temporal model for human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12116-12125, 2019.","title":"A neural temporal model for human motion prediction","authors":"Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, Alexander G Ororbia","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gui et al., 2018] Liang-Yan Gui, Yu-Xiong Wang, Deva Ra- manan, and José MF Moura. Few-shot human motion pre- diction via meta-learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 432-450, 2018.","title":"Few-shot human motion prediction via meta-learning","authors":"Liang-Yan, Yu-Xiong Gui, Deva Wang, José Mf Ramanan, Moura","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ionescu et al., 2013] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sens- ing in natural environments. IEEE transactions on pat- tern analysis and machine intelligence, 36(7):1325-1339, 2013.","title":"Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments","authors":"Catalin Ionescu, Dragos Papava, Vlad Olaru, Cristian Sminchisescu","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koch, 2015] Gregory Koch. Siamese neural networks for one-shot image recognition. In International Conference on Machine Learning, 2015.","title":"Siamese neural networks for one-shot image recognition","authors":"Gregory Koch","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koppula and Saxena, 2015] Hema S Koppula and Ashutosh Saxena. Anticipating human activities using object affor- dances for reactive robotic response. IEEE transactions on pattern analysis and machine intelligence, 38(1):14- 29, 2015.","title":"Anticipating human activities using object affordances for reactive robotic response","authors":"S Hema, Ashutosh Koppula, Saxena","year":"2015","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018] Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. Convolutional sequence to sequence model for human dynamics. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 5226-5234, 2018.","title":"Convolutional sequence to sequence model for human dynamics","authors":"Chen Li, Zhen Zhang, Wee Sun Lee, Gim Hee, Lee","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019] Zhenguang Liu, Shuang Wu, Shuyuan Jin, Qi Liu, Shijian Lu, Roger Zimmermann, and Li Cheng. Towards natural and accurate future motion prediction of humans and animals. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 10004-10012, 2019.","title":"Towards natural and accurate future motion prediction of humans and animals","authors":"Zhenguang Liu, Shuang Wu, Shuyuan Jin, Qi Liu, Shijian Lu, Roger Zimmermann, Li Cheng","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lopez-Paz and Ranzato, 2017] David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6467-6476, 2017.","title":"Gradient episodic memory for continual learning","authors":"David Lopez, - Paz, Marc'aurelio Ranzato","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Martinez et al., 2017] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2891-2900, 2017.","title":"On human motion prediction using recurrent neural networks","authors":"Julieta Martinez, Michael J Black, Javier Romero","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pavlovic et al., 2001] Vladimir Pavlovic, James M Rehg, and John MacCormick. Learning switching linear mod- els of human motion. In Advances in neural information processing systems, pages 981-987, 2001.","title":"Learning switching linear models of human motion","authors":"Vladimir Pavlovic, James M Rehg, John Maccormick","year":"2001","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ravi and Larochelle, 2016] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2016.","title":"Optimization as a model for few-shot learning","authors":"Larochelle ; Ravi, Sachin Ravi, Hugo Larochelle","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason We- ston, Rob Fergus, et al. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448, 2015.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2019] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 403-412, 2019.","title":"Meta-transfer learning for few-shot learning","authors":"Sun","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Taylor et al., 2007] Graham W Taylor, Geoffrey E Hinton, and Sam T Roweis. Modeling human motion using binary latent variables. In Advances in neural information pro- cessing systems, pages 1345-1352, 2007.","title":"Modeling human motion using binary latent variables","authors":"Geoffrey E Graham W Taylor, Sam T Hinton, Roweis","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim- othy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vondrak et al., 2008] Marek Vondrak, Leonid Sigal, and Odest Chadwicke Jenkins. Physical simulation for prob- abilistic motion tracking. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2008.","title":"Physical simulation for probabilistic motion tracking","authors":"Marek Vondrak, Leonid Sigal, Odest Chadwicke Jenkins","year":"2008","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2006] Jack Wang, Aaron Hertzmann, and David J Fleet. Gaussian process dynamical models. In Advances in neural information processing systems, pages 1441-1448, 2006.","title":"Gaussian process dynamical models","authors":"Wang","year":"2006","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2017] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pages 7029- 7039, 2017.","title":"Learning to model the tail","authors":"Yu-Xiong Wang, Deva Ramanan, Martial Hebert","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wei et al., 2019] Mao Wei, Liu Miaomiao, Salzemann Mathieu, and Li Hongdong. Learning trajectory dependen- cies for human motion prediction. In International Con- ference on Computer Vision, 2019.","title":"Learning trajectory dependencies for human motion prediction","authors":"Mao Wei, Liu Miaomiao, Salzemann Mathieu, Li Hongdong","year":"2019","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}