---
title: "Profiling the Voice: Speaker-Specific Phoneme Fingerprinting for Speech Deepfake Detection"
authors:
  - "Jun Xue"
  - "Tong Zhang"
  - "Zhuolin Yi"
  - "Yihuan Huang"
  - "Yi Chai"
  - "Yiyang Zhang"
  - "Yanzhen Ren"
doi: "10.24963/ijcai.2026/667"
url: "https://ijcai.org/proceedings/2026/667"
pdf: "https://ijcai.org/proceedings/2026/0667.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "5991-5999"
track: "Main Track"
keywords:
  - "Natural Language Processing: Speech"
---

# Profiling the Voice: Speaker-Specific Phoneme Fingerprinting for Speech Deepfake Detection

Jun Xue, Tong Zhang, Zhuolin Yi, Yihuan Huang, Yi Chai, Yiyang Zhang, Yanzhen Ren

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 5991-5999. DOI: [10.24963/ijcai.2026/667](https://doi.org/10.24963/ijcai.2026/667)

## Abstract

The rapid advancement of generative AI has made audio deepfakes increasingly indistinguishable from authentic human vocals, posing significant threats to persons-of-interest (POI) such as public figures. Current detection systems primarily rely on generic, black-box models that fail to capture speaker-specific idiosyncratic traits and lack interpretability. In this paper, we propose Phoneme-based Voice Profiling (PVP), a novel personalized defense framework. By shifting the detection paradigm from macro-utterance analysis to micro-phonetic modeling, PVP captures the unique acoustic distributions underlying a POI’s habitual articulatory patterns. Specifically, our framework models speaker-specific phonetic realizations using lightweight Gaussian Mixture Models (GMMs) estimated solely from bona fide reference speech. This design enables data-efficient profiling and robust generalization to previously unseen spoofing attacks without requiring heavy spoof-specific training. Furthermore, we introduce the first large-scale Chinese POI deepfake dataset to benchmark speaker-specific detection. Experimental results demonstrate that PVP significantly outperforms state-of-the-art generic detectors in POI spoofing scenarios, achieving substantial EER reductions while providing fine-grained, phoneme-level interpretability for forensic analysis. Code and data are available at: https://github.com/JunXue-tech/PVP