{"type":"ScholarlyArticle","title":"MotionMixer: MLP-based 3D Human Body Pose Forecasting","authors":[{"name":"Arij Bouazizi","slug":"arij-bouazizi","given":"Arij","family":"Bouazizi","affiliation":"Mercedes-Benz AG , Stuttgart , Germany; Ulm University , Ulm , Germany"},{"name":"Adrian Holzbock","slug":"adrian-holzbock","given":"Adrian","family":"Holzbock","affiliation":"Ulm University , Ulm , Germany"},{"name":"Ulrich Kressel","slug":"ulrich-kressel","given":"Ulrich","family":"Kressel","affiliation":"Mercedes-Benz AG , Stuttgart , Germany"},{"name":"Klaus Dietmayer","slug":"klaus-dietmayer","given":"Klaus","family":"Dietmayer","affiliation":"Ulm University , Ulm , Germany"},{"name":"Vasileios Belagiannis","slug":"vasileios-belagiannis","given":"Vasileios","family":"Belagiannis","affiliation":"Otto von Guericke University Magdeburg , Magdeburg , Germany"}],"abstract":"In this work, we present MotionMixer, an efficient 3D human body pose forecasting model based solely on multi-layer perceptrons (MLPs). MotionMixer learns the spatial-temporal 3D body pose dependencies by sequentially mixing both modalities. Given a stacked sequence of 3D body poses, a spatial-MLP extracts fine-grained spatial dependencies of the body joints. The interaction of the body joints over time is then modelled by a temporal MLP. The spatial-temporal mixed features are finally aggregated and decoded to obtain the future motion. To calibrate the influence of each time step in the pose sequence, we make use of squeeze-and-excitation (SE) blocks. We evaluate our approach on Human3.6M, AMASS, and 3DPW datasets using the standard evaluation protocols. For all evaluations, we demonstrate state-of-the-art performance, while having a model with a smaller number of parameters. Our code is available at: https://github.com/MotionMLP/MotionMixer.","doi":"10.24963/ijcai.2022/111","url":"https://ijcai.org/proceedings/2022/111","pdf":"https://ijcai.org/proceedings/2022/0111.pdf","markdown":"https://ijcai.org/proceedings/2022/111.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/111","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":111,"pages":{"first":791,"last":798},"keywords":["Computer Vision: Motion and Tracking","Computer Vision: Biometrics, Face, Gesture and Pose Recognition"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986956","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985099","length_sec":617,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aksan et al., 2020] Emre Aksan, Peng Cao, Manuel Kauf- mann, and Otmar Hilliges. A spatio-temporal trans- former for 3d human motion prediction. arXiv preprint arXiv:2004.08692, 2020.","title":"A spatio-temporal transformer for 3d human motion prediction","authors":"Emre Aksan, Peng Cao, Manuel Kaufmann, Otmar Hilliges","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Belagiannis et al., 2014] Vasileios Belagiannis, Christian Amann, Nassir Navab, and Slobodan Ilic. Holistic hu- man pose estimation with regression forests. In Interna- tional Conference on Articulated Motion and Deformable Objects, pages 20-30. Springer, 2014.","title":"Holistic human pose estimation with regression forests","authors":"Vasileios Belagiannis, Christian Amann, Nassir Navab, Slobodan Ilic","year":"2014","venue":"International Conference on Articulated Motion and Deformable Objects","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bouazizi et al., 2021] Arij Bouazizi, Julian Wiederer, Ul- rich Kressel, and Vasileios Belagiannis. Self-supervised 3d human pose estimation with multiple-view geometry. In 2021 16th IEEE International Conference on Auto- matic Face and Gesture Recognition (FG 2021), pages 1- 8. IEEE, 2021.","title":"Self-supervised 3d human pose estimation with multiple-view geometry","authors":"Arij Bouazizi, Julian Wiederer, Ulrich Kressel, Vasileios Belagiannis","year":"2021","venue":"16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dang et al., 2021] Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Msr-gcn: Multi-scale residual graph convolution networks for hu- man motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11467-11476, 2021.","title":"Msr-gcn: Multi-scale residual graph convolution networks for human motion prediction","authors":"Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, Guiqing Li","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fragkiadaki et al., 2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346-4354, 2015.","title":"Recurrent network models for human dynamics","authors":"Katerina Fragkiadaki, Sergey Levine, Panna Felsen, Jitendra Malik","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gui et al., 2018] Liang-Yan Gui, Kevin Zhang, Yu-Xiong Wang, Xiaodan Liang, José MF Moura, and Manuela Veloso. Teaching robots to predict human motion. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 562-567. IEEE, 2018.","title":"Teaching robots to predict human motion","authors":"Kevin Liang-Yan Gui, Yu-Xiong Zhang, Xiaodan Wang, José Mf Liang, Manuela Moura, Veloso","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.","title":"Gaussian error linear units","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ionescu et al., 2013] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sens- ing in natural environments. IEEE transactions on pat- tern analysis and machine intelligence, 36(7):1325-1339, 2013.","title":"Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments","authors":"Catalin Ionescu, Dragos Papava, Vlad Olaru, Cristian Sminchisescu","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulić et al., 2012] Dana Kulić, Christian Ott, Dongheui Lee, Junichi Ishikawa, and Yoshihiko Nakamura. Incre- mental learning of full body motion primitives and their sequencing through human motion observation. The In- ternational Journal of Robotics Research, 31(3):330-345, 2012.","title":"Incremental learning of full body motion primitives and their sequencing through human motion observation","authors":"Dana Kulić, Christian Ott, Dongheui Lee, Junichi Ishikawa, Yoshihiko Nakamura","year":"2012","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018] Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. Convolutional sequence to sequence model for human dynamics. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 5226-5234, 2018.","title":"Convolutional sequence to sequence model for human dynamics","authors":"Chen Li, Zhen Zhang, Wee Sun Lee, Gim Hee, Lee","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Zhenguang Liu, Pengxiang Su, Shuang Wu, Xuanjing Shen, Haipeng Chen, Yanbin Hao, and Meng Wang. Motion prediction using trajectory cues. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13299-13308, 2021.","title":"Motion prediction using trajectory cues","authors":"Liu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mahmood et al., 2019] Naureen Mahmood, Nima Ghor- bani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5442-5451, 2019.","title":"Amass: Archive of motion capture as surface shapes","authors":"Naureen Mahmood, Nima Ghorbani, Gerard Nikolaus F Troje, Michael J Pons-Moll, Black","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mao et al., 2019] Wei Mao, Miaomiao Liu, Mathieu Salz- mann, and Hongdong Li. Learning trajectory dependen- cies for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9489-9497, 2019.","title":"Learning trajectory dependencies for human motion prediction","authors":"Wei Mao, Miaomiao Liu, Mathieu Salzmann, Hongdong Li","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mao et al., 2020] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion predic- tion via motion attention. In European Conference on Computer Vision, pages 474-489. Springer, 2020.","title":"History repeats itself: Human motion prediction via motion attention","authors":"Wei Mao, Miaomiao Liu, Mathieu Salzmann","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mao et al., 2021] Wei Mao, Miaomiao Liu, Mathieu Salz- mann, and Hongdong Li. Multi-level motion attention for human motion prediction. International Journal of Com- puter Vision, pages 1-23, 2021.","title":"Multi-level motion attention for human motion prediction","authors":"Wei Mao, Miaomiao Liu, Mathieu Salzmann, Hongdong Li","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Martinez et al., 2017] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2891-2900, 2017.","title":"On human motion prediction using recurrent neural networks","authors":"Julieta Martinez, Michael J Black, Javier Romero","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sofianos et al., 2021] Theodoros Sofianos, Alessio Sampieri, Luca Franco, and Fabio Galasso. Space- time-separable graph convolutional network for pose forecasting. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 11209-11218, 2021.","title":"Spacetime-separable graph convolutional network for pose forecasting","authors":"Theodoros Sofianos, Alessio Sampieri, Luca Franco, Fabio Galasso","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2018] Yongyi Tang, Lin Ma, Wei Liu, and Wei- Shi Zheng. Long-term human motion prediction by mod- eling motion context and enhancing motion dynamics. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 935-941. International Joint Conferences on Artificial In- telligence Organization, 7 2018.","title":"Long-term human motion prediction by modeling motion context and enhancing motion dynamics","authors":"Yongyi Tang, Lin Ma, Wei Liu, Wei-Shi Zheng","year":"2018","venue":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18","doi":null,"url":null,"internal_url":"/proceedings/2018/130"},{"position":22,"raw":"[Tolstikhin et al., 2021] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.","title":"Mlp-mixer: An all-mlp architecture for vision","authors":"Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit","year":"2021","venue":"Thirty-Fifth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[von Marcard et al., 2018] Timo von Marcard, Roberto Hen- schel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European Conference on Computer Vision (ECCV), pages 601-617, 2018.","title":"Recovering accurate 3d human pose in the wild using imus and a moving camera","authors":"Von Marcard","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2007] Jack M Wang, David J Fleet, and Aaron Hertzmann. Gaussian process dynamical models for hu- man motion. IEEE transactions on pattern analysis and machine intelligence, 30(2):283-298, 2007.","title":"Gaussian process dynamical models for human motion","authors":"M Jack, David J Wang, Aaron Fleet, Hertzmann","year":"2007","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wiederer et al., 2020] Julian Wiederer, Arij Bouazizi, Ul- rich Kressel, and Vasileios Belagiannis. Traffic control gesture recognition for autonomous vehicles. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10676-10683. IEEE, 2020.","title":"Traffic control gesture recognition for autonomous vehicles","authors":"Julian Wiederer, Arij Bouazizi, Ulrich Kressel, Vasileios Belagiannis","year":"2020","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhong et al., 2022] Chongyang Zhong, Lei Hu, Zihao Zhang, Yongjing Ye, and Shihong Xia. Spatial-temporal gating-adjacency gcn for human motion prediction. arXiv preprint arXiv:2203.01474, 2022.","title":"Spatial-temporal gating-adjacency gcn for human motion prediction","authors":"Chongyang Zhong, Lei Hu, Zihao Zhang, Yongjing Ye, Shihong Xia","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}