{"type":"ScholarlyArticle","title":"ReliaAvatar: A Robust Real-Time Avatar Animator with Integrated Motion Prediction","authors":[{"name":"Bo Qian","slug":"bo-qian","given":"Bo","family":"Qian","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Zhenhuan Wei","slug":"zhenhuan-wei","given":"Zhenhuan","family":"Wei","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Jiashuo Li","slug":"jiashuo-li","given":"Jiashuo","family":"Li","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Xing Wei","slug":"xing-wei","given":"Xing","family":"Wei","affiliation":"School of Software Engineering , Xi'an Jiaotong University"}],"abstract":"Efficiently estimating the full-body pose with minimal wearable devices presents a worthwhile research direction.  Despite significant advancements in this field, most current research neglects to explore full-body avatar estimation under low-quality signal conditions, which is prevalent in practical usage. To bridge this gap, we summarize three scenarios that may be encountered in real-world applications: standard scenario, instantaneous data-loss scenario, and prolonged data-loss scenario, and propose a new evaluation benchmark. The solution we propose to address data-loss scenarios is integrating the full-body avatar pose estimation problem with motion prediction. Specifically, we present ReliaAvatar, a real-time, reliable avatar animator equipped with predictive modeling capabilities employing a dual-path architecture. ReliaAvatar operates effectively, with an impressive performance rate of 109 frames per second (fps). Extensive comparative evaluations on widely recognized benchmark datasets demonstrate ReliaAvatar's superior performance in both standard and low data-quality conditions. The code is available at https://github.com/MIV-XJTU/ReliaAvatar.","doi":"10.24963/ijcai.2024/346","url":"https://ijcai.org/proceedings/2024/346","pdf":"https://ijcai.org/proceedings/2024/0346.pdf","markdown":"https://ijcai.org/proceedings/2024/346.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/346","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Humans and AI","track":"Main Track","number":346,"pages":{"first":3124,"last":3132},"keywords":["Robotics: ROB: Human robot interaction","Humans and AI: HAI: Human-computer interaction","Humans and AI: HAI: Personalization and user modeling","Humans and AI: HAI: Applications"],"videos":[],"references":[{"position":1,"raw":"[Ahuja et al., 2021] Karan Ahuja, Eyal Ofek, Mar Gonzalez- Franco, Christian Holz, and Andrew D Wilson. Cool- moves: User motion accentuation in virtual reality. Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(2):1-23, 2021.","title":"Coolmoves: User motion accentuation in virtual reality","authors":"Karan Ahuja, Eyal Ofek, Mar Gonzalez-Franco, Christian Holz, Andrew D Wilson","year":"2021","venue":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aliakbarian et al., 2023] Sadegh Aliakbarian, Fatemeh Saleh, David Collier, Pashmina Cameron, and Darren Cosker. Hmd-nemo: Online 3d avatar motion gener- ation from sparse observations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9622-9631, 2023.","title":"Hmd-nemo: Online 3d avatar motion generation from sparse observations","authors":"Fatemeh Sadegh Aliakbarian, David Saleh, Pashmina Collier, Darren Cameron, Cosker","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2024] Yifan Bai, Zeyang Zhao, Yihong Gong, and Xing Wei. Artrackv2: Prompting autoregressive tracker where to look and how to describe. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.","title":"Artrackv2: Prompting autoregressive tracker where to look and how to describe","authors":"Yifan Bai, Zeyang Zhao, Yihong Gong, Xing Wei","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Di et al., 2023] Xinhan Di, Xiaokun Dai, Xinkang Zhang, and Xinrong Chen. Dual attention poser: Dual path body tracking based on attention. In 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Work- shops (CVPRW), pages 2795-2804. IEEE, 2023.","title":"Dual attention poser: Dual path body tracking based on attention","authors":"Di","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dittadi et al., 2021] Andrea Dittadi, Sebastian Dziadzio, Darren Cosker, Ben Lundell, Thomas J Cashman, and Jamie Shotton. Full-body motion from a single head- mounted device: Generating smpl poses from partial ob- servations. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 11687- 11697, 2021.","title":"Full-body motion from a single headmounted device: Generating smpl poses from partial observations","authors":"Andrea Dittadi, Sebastian Dziadzio, Darren Cosker, Ben Lundell, Thomas J Cashman, Jamie Shotton","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Du et al., 2023] Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 481-490, 2023. [graphics lab., 2000] CMU graphics lab. Cmu graphics lab motion capture database. http://mocap.cs.cmu.edu/, 2000. Accessed: 2024-05-08.","title":"Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model","authors":"Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, Artsiom Sanakoyeu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":"http://mocap.cs.cmu.edu/,2000","internal_url":null},{"position":8,"raw":"[Huang et al., 2018] Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J Black, Otmar Hilliges, and Ger- ard Pons-Moll. Deep inertial poser: Learning to recon- struct human pose from sparse inertial measurements in real time. ACM Transactions on Graphics (TOG), 37(6):1- 15, 2018.","title":"Deep inertial poser: Learning to reconstruct human pose from sparse inertial measurements in real time","authors":"Huang","year":"2018","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jain et al., 2016] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In Proceedings of the ieee conference on computer vision and pattern recogni- tion, pages 5308-5317, 2016.","title":"Structural-rnn: Deep learning on spatio-temporal graphs","authors":"Ashesh Jain, Silvio Amir R Zamir, Ashutosh Savarese, Saxena","year":"2016","venue":"Proceedings of the ieee conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2022] Jiaxi Jiang, Paul Streli, Huajian Qiu, An- dreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. In European Conference on Computer Vision, pages 443-460. Springer, 2022.","title":"Avatarposer: Articulated full-body pose tracking from sparse motion sensing","authors":"Jiang","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2023] Jiaxi Jiang, Paul Streli, Manuel Meier, Andreas Fender, and Christian Holz. Egoposer: Robust real-time ego-body pose estimation in large scenes. arXiv preprint arXiv:2308.06493, 2023.","title":"Egoposer: Robust real-time ego-body pose estimation in large scenes","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lehrmann et al., 2014] Andreas M Lehrmann, Peter V Gehler, and Sebastian Nowozin. Efficient nonlinear markov models for human motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 1314-1321, 2014.","title":"Efficient nonlinear markov models for human motion","authors":"Peter V Andreas M Lehrmann, Sebastian Gehler, Nowozin","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2019] Zhenguang Liu, Shuang Wu, Shuyuan Jin, Qi Liu, Shijian Lu, Roger Zimmermann, and Li Cheng. Towards natural and accurate future motion prediction of humans and animals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10004-10012, 2019.","title":"Towards natural and accurate future motion prediction of humans and animals","authors":"Zhenguang Liu, Shuang Wu, Shuyuan Jin, Qi Liu, Shijian Lu, Roger Zimmermann, Li Cheng","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2022] Tiezheng Ma, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Progressively gen- erating better initial guesses towards next stages for high- quality human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6437-6446, 2022.","title":"Progressively generating better initial guesses towards next stages for highquality human motion prediction","authors":"Ma","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mahmood et al., 2019] Naureen Mahmood, Nima Ghor- bani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442-5451, 2019.","title":"Amass: Archive of motion capture as surface shapes","authors":"Naureen Mahmood, Nima Ghorbani, Gerard Nikolaus F Troje, Michael J Pons-Moll, Black","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mao et al., 2019] Wei Mao, Miaomiao Liu, Mathieu Salz- mann, and Hongdong Li. Learning trajectory dependen- cies for human motion prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9489-9497, 2019.","title":"Learning trajectory dependencies for human motion prediction","authors":"Wei Mao, Miaomiao Liu, Mathieu Salzmann, Hongdong Li","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mao et al., 2020] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion pre- diction via motion attention. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 474-489. Springer, 2020.","title":"History repeats itself: Human motion prediction via motion attention","authors":"Wei Mao, Miaomiao Liu, Mathieu Salzmann","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Martinez et al., 2017] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using re- current neural networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2891-2900, 2017.","title":"On human motion prediction using recurrent neural networks","authors":"Julieta Martinez, Michael J Black, Javier Romero","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Müller et al., 2007] Meinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Björn Krüger, and Andreas Weber. Mocap database hdm05. Institut für Informatik II, Universität Bonn, 2(7), 2007.","title":"Mocap database hdm05","authors":"Müller","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shi et al., 2022] Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan. End-to-end multi-person pose estima- tion with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11069-11078, June 2022.","title":"End-to-end multi-person pose estimation with transformers","authors":"Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, Wenming Tan","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Taylor et al., 2006] Graham W Taylor, Geoffrey E Hinton, and Sam Roweis. Modeling human motion using binary latent variables. Advances in neural information process- ing systems, 19, 2006.","title":"Modeling human motion using binary latent variables","authors":"Geoffrey E Graham W Taylor, Sam Hinton, Roweis","year":"2006","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Troje, 2002] Nikolaus F Troje. Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of vision, 2(5):2-2, 2002.","title":"Decomposing biological motion: A framework for analysis and synthesis of human gait patterns","authors":"Nikolaus F Troje","year":"2002","venue":"Journal of vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Unity, 2018] Unity. The final inverse kinematics solu- tion for unity. https://assetstore.unity.com/packages/tools/ animation/final-ik-14290, 2018. Accessed: 2024-05-08.","title":"Unity. The final inverse kinematics solution for unity","authors":null,"year":"2018","venue":null,"doi":null,"url":"https://assetstore.unity.com/packages/tools/animation/final-ik-14290,2018","internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vlasic et al., 2007] Daniel Vlasic, Rolf Adelsberger, Gio- vanni Vannucci, John Barnwell, Markus Gross, Wojciech Matusik, and Jovan Popović. Practical motion capture in everyday surroundings. ACM transactions on graphics (TOG), 26(3):35-es, 2007.","title":"Practical motion capture in everyday surroundings","authors":"Daniel Vlasic, Rolf Adelsberger, Giovanni Vannucci, John Barnwell, Markus Gross, Wojciech Matusik, Jovan Popović","year":"2007","venue":"ACM transactions on graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Von Marcard et al., 2017] Timo Von Marcard, Bodo Rosen- hahn, Michael J Black, and Gerard Pons-Moll. Sparse in- ertial poser: Automatic 3d human pose estimation from sparse imus. In Computer graphics forum, volume 36, pages 349-360. Wiley Online Library, 2017.","title":"Sparse inertial poser: Automatic 3d human pose estimation from sparse imus","authors":"Von Timo, Bodo Marcard, Michael J Rosenhahn, Gerard Black, Pons-Moll","year":"2017","venue":"Computer graphics forum","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wei et al., 2023] Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual track- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 9697-9706, 2023.","title":"Autoregressive visual tracking","authors":"Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, Yihong Gong","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xsens, 2013] MVN Xsens. Full 6dof human motion track- ing using miniature inertial sensors. Daniel Roetenber- gLuingeHenk, 2013.","title":"MVN Xsens. Full 6dof human motion tracking using miniature inertial sensors","authors":"Xsens","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yan et al., 2024] Kangwei Yan, Fei Wang, Bo Qian, Han Ding, Jinsong Han, and Xing Wei. Person-in-wifi 3d: End- to-end multi-person 3d pose estimation with wi-fi. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.","title":"Person-in-wifi 3d: Endto-end multi-person 3d pose estimation with wi-fi","authors":"Kangwei Yan, Fei Wang, Bo Qian, Han Ding, Jinsong Han, Xing Wei","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2021] Dongseok Yang, Doyeon Kim, and Sung- Hee Lee. Lobstr: Real-time lower-body pose prediction from sparse upper-body tracking signals. In Computer Graphics Forum, volume 40, pages 265-275. Wiley On- line Library, 2021.","title":"Lobstr: Real-time lower-body pose prediction from sparse upper-body tracking signals","authors":"Yang","year":"2021","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yi et al., 2021] Xinyu Yi, Yuxiao Zhou, and Feng Xu. Transpose: Real-time 3d human translation and pose es- timation with six inertial sensors. ACM Transactions on Graphics (TOG), 40(4):1-13, 2021.","title":"Transpose: Real-time 3d human translation and pose es-timation with six inertial sensors","authors":"Xinyu Yi, Yuxiao Zhou, Feng Xu","year":"2021","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zheng et al., 2023] Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, and Xiaojie Jin. Realistic full-body tracking from sparse observations via joint-level modeling. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 14678-14688, 2023.","title":"Realistic full-body tracking from sparse observations via joint-level modeling","authors":"Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, Xiaojie Jin","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745-5753, 2019.","title":"On the continuity of rotation representations in neural networks","authors":"Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, Hao Li","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}