{"type":"ScholarlyArticle","title":"SyncGaussian: Stable 3D Gaussian-Based Talking Head Generation with Enhanced Lip Sync via Discriminative Speech Features","authors":[{"name":"Ke Liu","slug":"ke-liu","given":"Ke","family":"Liu","affiliation":""},{"name":"Jiwei Wei","slug":"jiwei-wei","given":"Jiwei","family":"Wei","affiliation":""},{"name":"Shiyuan He","slug":"shiyuan-he","given":"Shiyuan","family":"He","affiliation":""},{"name":"Zeyu Ma","slug":"zeyu-ma","given":"Zeyu","family":"Ma","affiliation":""},{"name":"Chaoning Zhang","slug":"chaoning-zhang","given":"Chaoning","family":"Zhang","affiliation":""},{"name":"Ning Xie","slug":"ning-xie","given":"Ning","family":"Xie","affiliation":""},{"name":"Yang Yang","slug":"yang-yang","given":"Yang","family":"Yang","affiliation":""}],"abstract":"Generating high-fidelity talking heads that maintain stable head poses and achieve robust lip sync remains a significant challenge. Although methods based on 3D Gaussian Splatting (3DGS) offer a promising solution via point-based deformation, they suffer from inconsistent head dynamics and mismatched mouth movements due to unstable Gaussian initialization and incomplete speech features. To overcome these limitations, we introduce SyncGaussian, a 3DGS-based framework that ensures stable head poses, enhanced lip sync, and realistic appearances with real-time rendering. SyncGaussian employs a stable head Gaussian initialization strategy to mitigate head jitter by optimizing commonly used rough head pose parameters. To enhance lip sync, we propose a sync-enhanced encoder that leverages audio-to-text and audio-to-visual speech features. Guided by a tailored cosine similarity loss function, the encoder integrates discriminative speech features through a multi-level sync adaptation mechanism, enabling the learning of an adaptive speech feature space. Extensive experiments demonstrate that SyncGaussian outperforms state-of-the-art methods in image quality, dynamic motion, and lip sync, with the potential for real-time applications.","doi":"10.24963/ijcai.2025/176","url":"https://ijcai.org/proceedings/2025/176","pdf":"https://ijcai.org/proceedings/2025/0176.pdf","markdown":"https://ijcai.org/proceedings/2025/176.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/176","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Computer Vision","track":"Main Track","number":176,"pages":{"first":1576,"last":1584},"keywords":["Computer Vision: CV: Image and video synthesis and generation","Computer Vision: CV: Image and video retrieval","Humans and AI: HAI: Applications","Computer Vision: CV: Multimodal learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}