{"type":"ScholarlyArticle","title":"Text/Speech-Driven Full-Body Animation","authors":[{"name":"Wenlin Zhuang","slug":"wenlin-zhuang","given":"Wenlin","family":"Zhuang","affiliation":"XR Lab , Alibaba Group"},{"name":"Jinwei Qi","slug":"jinwei-qi","given":"Jinwei","family":"Qi","affiliation":"XR Lab , Alibaba Group"},{"name":"Peng Zhang","slug":"peng-zhang","given":"Peng","family":"Zhang","affiliation":"XR Lab , Alibaba Group"},{"name":"Bang Zhang","slug":"bang-zhang","given":"Bang","family":"Zhang","affiliation":"XR Lab , Alibaba Group"},{"name":"Ping Tan","slug":"ping-tan","given":"Ping","family":"Tan","affiliation":"XR Lab , Alibaba Group"}],"abstract":"Due to the increasing demand in films and games, synthesizing 3D avatar animation has attracted much attention recently. In this work, we present a production-ready text/speech-driven full-body animation synthesis system. Given the text and corresponding speech, our system synthesizes face and body animations simultaneously, which are then skinned and rendered to obtain a video stream output. We adopt a learning-based approach for synthesizing facial animation and a graph-based approach to animate the body, which generates  high-quality avatar animation efficiently and robustly. Our results demonstrate the generated avatar animations are realistic, diverse and highly text/speech-correlated.","doi":"10.24963/ijcai.2022/863","url":"https://ijcai.org/proceedings/2022/863","pdf":"https://ijcai.org/proceedings/2022/0863.pdf","markdown":"https://ijcai.org/proceedings/2022/863.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/863","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Demo Track","track":"Demo Track","number":863,"pages":{"first":5956,"last":5959},"keywords":["Humans and AI: Cognitive Modeling","Humans and AI: Cognitive Systems","Agent-based and Multi-agent Systems: Human-Agent Interaction","Humans and AI: Human-Computer Interaction","Humans and AI: Applications"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987213","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985356","length_sec":340,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alexanderson et al., 2020] Simon Alexanderson, Gus- tav Eje Henter, Taras Kucherenko, and Jonas Beskow. Style-controllable speech-driven gesture synthesis us- ing normalising flows. In Computer Graphics Forum, volume 39, pages 487-496. Wiley Online Library, 2020.","title":"Style-controllable speech-driven gesture synthesis using normalising flows","authors":"Gustav Eje Simon Alexanderson, Taras Henter, Jonas Kucherenko, Beskow","year":"2020","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talk- ing face generation with dynamic pixel-wise loss. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7832-7841, 2019.","title":"Hierarchical cross-modal talking face generation with dynamic pixel-wise loss","authors":"Lele Chen, Ross K Maddox, Zhiyao Duan, Chenliang Xu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Kang Chen, Zhipeng Tan, Jin Lei, Song- Hai Zhang, Yuan-Chen Guo, Weidong Zhang, and Shi- Min Hu. Choreomaster: choreography-oriented music- driven dance synthesis. ACM Transactions on Graphics (TOG), 40(4):1-13, 2021.","title":"Choreomaster: choreography-oriented musicdriven dance synthesis","authors":"Kang Chen, Zhipeng Tan, Jin Lei, Song-Hai Zhang, Yuan-Chen Guo, Weidong Zhang, Shi-Min Hu","year":"2021","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ferstl et al., 2021] Ylva Ferstl, Michael Neff, and Rachel McDonnell. Expressgesture: Expressive gesture genera- tion from speech through database matching. Computer Animation and Virtual Worlds, 32(3-4):e2016, 2021.","title":"Expressgesture: Expressive gesture generation from speech through database matching","authors":"Ferstl","year":"2021","venue":"Computer Animation and Virtual Worlds","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ginosar et al., 2019] Shiry Ginosar, Amir Bar, Gefen Ko- havi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3497-3506, 2019.","title":"Learning individual styles of conversational gesture","authors":"Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, Jitendra Malik","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2021a] Li Hu, Jinwei Qi, Bang Zhang, Pan Pan, and Yinghui Xu. Text-driven 3d avatar animation with emotional and expressive behaviors. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2816-2818, 2021.","title":"Text-driven 3d avatar animation with emotional and expressive behaviors","authors":"Hu","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2021b] Li Hu, Bang Zhang, Peng Zhang, Jinwei Qi, Jian Cao, Daiheng Gao, Haiming Zhao, Xiaoduan Feng, Qi Wang, Lian Zhuo, et al. A virtual character gen- eration and animation system for e-commerce live stream- ing. In Proceedings of the 29th ACM International Con- ference on Multimedia, pages 1202-1211, 2021.","title":"A virtual character generation and animation system for e-commerce live streaming","authors":"Hu","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karras et al., 2017] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial an- imation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG), 36(4):1-12, 2017.","title":"Audio-driven facial animation by joint end-to-end learning of pose and emotion","authors":"Tero Karras, Timo Aila, Samuli Laine, Antti Herva, Jaakko Lehtinen","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kovar et al., 2008] Lucas Kovar, Michael Gleicher, and Frédéric Pighin. Motion graphs. In ACM SIGGRAPH 2008 classes, pages 1-10. 2008.","title":"Motion graphs","authors":"Lucas Kovar, Michael Gleicher, Frédéric Pighin","year":"2008","venue":"ACM SIGGRAPH 2008 classes","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2019] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing to music. Advances in Neural Information Processing Systems, 32, 2019.","title":"Dancing to music","authors":"Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science con- ference, volume 8, pages 18-25. Citeseer, 2015.","title":"librosa: Audio and music signal analysis in python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang, P Daniel, Matt Ellis, Eric Mcvicar, Oriol Battenberg, Nieto","year":"2015","venue":"Proceedings of the 14th python in science conference","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Oh et al., 2019] Tae-Hyun Oh, Tali Dekel, Changil Kim, In- bar Mosseri, William T Freeman, Michael Rubinstein, and Wojciech Matusik. Speech2face: Learning the face be- hind a voice. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7539- 7548, 2019.","title":"Speech2face: Learning the face behind a voice","authors":"Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, Michael William T Freeman, Wojciech Rubinstein, Matusik","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shlizerman et al., 2018] Eli Shlizerman, Lucio Dery, Hay- den Schoen, and Ira Kemelmacher-Shlizerman. Audio to body dynamics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7574- 7583, 2018.","title":"Audio to body dynamics","authors":"Shlizerman","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wiles et al., 2018] Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face gener- ation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV), pages 670-686, 2018.","title":"X2face: A network for controlling face generation using images, audio, and pose codes","authors":"Olivia Wiles, Andrew Koepke, Zisserman","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}