ExpTalk: Diverse Emotional Expression via Adaptive Disentanglement and Refined Alignment for Speech-Driven 3D Facial Animation

Qu, Zhan; Zhang, Shengyu; Li, Mengze; Chen, Zhuo; Lv, Chengfei; Zhao, Zhou; Wu, Fei

doi:10.24963/ijcai.2025/202

ExpTalk: Diverse Emotional Expression via Adaptive Disentanglement and Refined Alignment for Speech-Driven 3D Facial Animation

Zhan Qu, Shengyu Zhang, Mengze Li, Zhuo Chen, Chengfei Lv, Zhou Zhao, Fei Wu

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 1811-1819.

https://doi.org/10.24963/ijcai.2025/202

Keywords

Computer Vision: CV: 3D computer vision
Agent-based and Multi-agent Systems: MAS: Human-agent interaction
Machine Learning: ML: Generative models
Computer Vision: CV: Multimodal learning

Abstract

Speech-driven 3D facial animation aims to create lifelike facial expressions that synchronize accurately with speech. Despite significant progress, many existing methods may focus on generating facial animation with a fixed emotional state, neglecting the diverse transformations of facial emotions under a given speech input. To solve this issue, we focus on exploring the refined alignment between speech representations and multiple domains in facial expression information. We aim to disentangle the spoken language and emotion facial priors from speech expressions, to guide the refinement of the facial vertices based on speech. To accomplish this objective, we propose ExpTalk, which first applies an Adaptive Disentanglement Variational Autoencoder (AD-VAE) to decouple facial expression aligned with spoken language and emotions of speech through contrastive learning. Then a Refined Alignment Diffusion (RAD) is employed to iteratively refine the decoupled facial expression priors through diffusion-based perturbations, producing facial animations that align with the emotional variations of the given speech. Extensive experiments prove the effectiveness of our ExpTalk by surpassing state-of-the-arts by a large margin.

Zhan Qu, Shengyu Zhang, Mengze Li, Zhuo Chen, Chengfei Lv, Zhou Zhao, and Fei Wu. 2025. ExpTalk: Diverse Emotional Expression via Adaptive Disentanglement and Refined Alignment for Speech-Driven 3D Facial Animation. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 1811–1819. https://doi.org/10.24963/ijcai.2025/202

@inproceedings{ijcai2025p202,
  title     = {ExpTalk: Diverse Emotional Expression via Adaptive Disentanglement and Refined Alignment for Speech-Driven 3D Facial Animation},
  author    = {Qu, Zhan and Zhang, Shengyu and Li, Mengze and Chen, Zhuo and Lv, Chengfei and Zhao, Zhou and Wu, Fei},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {1811--1819},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/202},
  url       = {https://doi.org/10.24963/ijcai.2025/202},
}

TY  - CPAPER
TI  - ExpTalk: Diverse Emotional Expression via Adaptive Disentanglement and Refined Alignment for Speech-Driven 3D Facial Animation
AU  - Qu, Zhan
AU  - Zhang, Shengyu
AU  - Li, Mengze
AU  - Chen, Zhuo
AU  - Lv, Chengfei
AU  - Zhao, Zhou
AU  - Wu, Fei
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 1811
EP  - 1819
DO  - 10.24963/ijcai.2025/202
UR  - https://doi.org/10.24963/ijcai.2025/202
ER  -