{"type":"ScholarlyArticle","title":"Profiling the Voice: Speaker-Specific Phoneme Fingerprinting for Speech Deepfake Detection","authors":[{"name":"Jun Xue","slug":"jun-xue","given":"Jun","family":"Xue","affiliation":""},{"name":"Tong Zhang","slug":"tong-zhang","given":"Tong","family":"Zhang","affiliation":""},{"name":"Zhuolin Yi","slug":"zhuolin-yi","given":"Zhuolin","family":"Yi","affiliation":""},{"name":"Yihuan Huang","slug":"yihuan-huang","given":"Yihuan","family":"Huang","affiliation":""},{"name":"Yi Chai","slug":"yi-chai","given":"Yi","family":"Chai","affiliation":""},{"name":"Yiyang Zhang","slug":"yiyang-zhang","given":"Yiyang","family":"Zhang","affiliation":""},{"name":"Yanzhen Ren","slug":"yanzhen-ren","given":"Yanzhen","family":"Ren","affiliation":""}],"abstract":"The rapid advancement of generative AI has made audio deepfakes increasingly indistinguishable from authentic human vocals, posing significant threats to persons-of-interest (POI) such as public figures. Current detection systems primarily rely on generic, black-box models that fail to capture speaker-specific idiosyncratic traits and lack interpretability. In this paper, we propose Phoneme-based Voice Profiling (PVP), a novel personalized defense framework. By shifting the detection paradigm from macro-utterance analysis to micro-phonetic modeling, PVP captures the unique acoustic distributions underlying a POI’s habitual articulatory patterns. Specifically, our framework models speaker-specific phonetic realizations using lightweight Gaussian Mixture Models (GMMs) estimated solely from bona fide reference speech. This design enables data-efficient profiling and robust generalization to previously unseen spoofing attacks without requiring heavy spoof-specific training. Furthermore, we introduce the first large-scale Chinese POI deepfake dataset to benchmark speaker-specific detection. Experimental results demonstrate that PVP significantly outperforms state-of-the-art generic detectors in POI spoofing scenarios, achieving substantial EER reductions while providing fine-grained, phoneme-level interpretability for forensic analysis. Code and data are available at: https://github.com/JunXue-tech/PVP","doi":"10.24963/ijcai.2026/667","url":"https://ijcai.org/proceedings/2026/667","pdf":"https://ijcai.org/proceedings/2026/0667.pdf","markdown":"https://ijcai.org/proceedings/2026/667.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/667","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Natural Language Processing","track":"Main Track","number":667,"pages":{"first":5991,"last":5999},"keywords":["Natural Language Processing: Speech"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}