Profiling the Voice: Speaker-Specific Phoneme Fingerprinting for Speech Deepfake Detection

1 Introduction

With the rapid development of artificial intelligence generated content (AIGC), speech synthesis and voice conversion technologies have achieved unprecedented realism, making speech deepfakes increasingly accessible and convincing. As a result, speech deepfake detection has become a critical research topic, especially in high-stakes scenarios such as judicial forensics, governmental communications, and public statements by celebrities or other well-known figures. In phoneme realizations are shaped by individual vocal characteristics and habitual articulation. By extracting speaker-unique phonemes and constructing "Phoneme Fingerprints" from reference speech, the system conducts granular verification on test speech and uncover "Phonetic Anomaly Cues", pinpointing specific suspicious phonetic segments for enhanced forensic transparency.

∗Corresponding author.

these contexts, a single forged utterance attributed to a specific speaker may lead to severe social, legal, or economic consequences. Therefore, beyond generic detection systems, there is an urgent need for personalized, speaker-specific, and interpretable deepfake speech protection mechanisms.

Most existing studies [Zhang et al., 2025a; Xue et al., 2026; Xue et al., 2023; Xue et al., 2024] formulate speech deepfake detection as a straightforward classification problem, where an entire speech segment is processed by an end-to-end black-box model to output utterance-level decision. A common paradigm is to extract front-end representations using self-supervised learning (SSL) models, followed by a complex back-end classifier to perform discrimination (Fig 1a). Representative works [Tak et al., 2022; Zhang et al., 2024; Tran et al., 2025] adopt convolutional or transformer-based architectures to model global acoustic patterns and have achieved impressive performance on widely used benchmark datasets[Todisco et al., 2019; M¨uller et al.,

2022].

However, such evaluations are typically conducted under speaker-agnostic settings, where training and test speakers are loosely constrained. As a result, the learned decision boundaries tend to emphasize dataset-level artifacts-such as synthesis-model-specific spectral patterns, vocoder traces, or corpus-dependent biases-that are consistent across speakers but largely unrelated to individual articulation habits. In contrast, in POI scenarios, attackers often explicitly tailor the synthesis process to a single target speaker by fine-tuning generative models with bona fide speech from that individual [Ali et al., 2025]. Under such conditions, generic detectors often struggle to capture subtle, speaker-dependent inconsistencies and provide limited interpretability regarding why a given utterance is detected as fake.

To address this limitation, we turn our attention to the phoneme. Compared to frame-level artifacts that are often transient and model-dependent, phonemes offer a linguistically grounded unit whose acoustic realizations exhibit stable, speaker-specific patterns. Although phoneme categories are shared across speakers, their realizations are shaped by individual vocal characteristics and habitual articulation behaviors, leading to consistent inter-speaker differences. These fine-grained regularities are difficult for current speech synthesis and voice cloning systems to reproduce faithfully, often giving rise to systematic phoneme-level inconsistencies. While recent studies [Zhang et al., 2025b; Baser et al., 2025] have begun to exploit phoneme-related information to guide deepfake detection models, phonemes are still treated as auxiliary signals within speaker-agnostic, utterance-level classifiers. Such designs overlook the fact that phoneme realizations are inherently speaker-dependent and physiologically constrained, limiting their ability to support personalized analysis and interpretable detection in personof-interest (POI) scenarios.

This intuition is further supported by a preliminary analysis on the FamousFigures dataset. We sample genuine speech from ten speakers and extract phoneme-related features using the MMS model [Ge et al., 2025], a multilingual speech deepfake detection system. These features are then visualized as heatmaps. As shown in Fig 2, distinct phoneme pronunciation patterns can be observed across different speakers, indicating that phoneme-level representations capture stable and discriminative speaker-specific characteristics. These observations suggest that phonemes provide a natural and interpretable basis for personalized modeling in speech deepfake detection.

Building on these insights, we propose a novel, plug-andplay phoneme-level modeling framework for speaker-specific deepfake detection (Fig. 1b). Given a small amount of bona fide reference speech from a POI, our method first performs phoneme-level alignment and extracts phoneme-conditioned acoustic representations. For each phoneme, a lightweight Gaussian Mixture Model (GMM) is constructed to characterize the speaker's habitual articulatory distribution. At inference time, a test utterance is decomposed into phonetic units and evaluated against the corresponding phoneme models, producing fine-grained consistency scores that reflect how well each phoneme realization matches the target speaker's profile. These phoneme-level consistency scores expose localized speaker-dependent deviations, offering fine-grained and interpretable evidence for deepfake analysis.

Distinct from existing heavy-training paradigms, our method is agnostic to synthesis algorithms, thus reduces overfitting to spoof-specific artifacts, and generalizes naturally to unseen spoofing attacks. Moreover, its additive nature allows it to function as an interpretable "plugin" that can be seamlessly combined with existing detection backbones. Our contributions can be summarized as:

• Speaker-Specific Phoneme Profiling: We propose a personalized deepfake detection framework that explicitly captures speaker-specific articulatory patterns using lightweight statistical modeling, enabling effective POI protection with minimal bona fide reference data and without relying on spoof-specific training.
• Fine-Grained and Interpretable Phonetic Evidence: We introduce an interpretable phoneme-level scoring and tiered decision mechanism that reveals explicit phonetic anomaly cues, facilitating forensic analysis and transparent reasoning beyond black-box utterance-level classifiers.
• Chinese POI Deepfake Dataset: To fill the gap in speaker-centric benchmarks for Mandarin, we present the first large-scale Chinese POI deepfake dataset. It covers diverse public figures and modern speech generation techniques, enabling systematic evaluation of personalized spoofing attacks.

2 RelatedWork

2.1 Self-Supervised Learning for Speech Deepfake Detection

The paradigm of speech deepfake detection has shifted significantly from handcrafted features [Xue et al., 2022;

Lavrentyeva et al., 2019] (e.g., LFCC, CQCC) to representation learning, driven by the success of Self-Supervised Learning (SSL) foundation models. Models such as wav2vec 2.0 [Baevski et al., 2020], HuBERT [Hsu et al., 2021], and WavLM [Chen et al., 2022] leverage massive unlabeled corpora to encode rich acoustic and phonetic information in a latent and entangled manner, proving highly effective at capturing subtle artifacts introduced by neural vocoders and upsampling layers. While Wav2Vec 2.0 utilizes contrastive predictive coding to retain phase information crucial for detecting synthesis discontinuities, WavLM explicitly incorporates a denoising objective, offering superior robustness against channel variation and background noise. Recent studies have further explored cross-lingual generalization using models like XLS-R [Babu et al., 2021] to address the scarcity of non-English benchmarks, particularly for tonal languages like Mandarin. However, most SSL-based detectors utilize these embeddings as global, utterance-level representations, often discarding the fine-grained, localized phonetic discrepancies that are critical for identifying high-fidelity, targeted spoofing attacks.

2.2 Speech Deepfake Detection: FromAgnostic to Speaker-Specific

Existing detection frameworks can be broadly categorized into speaker-agnostic and speaker-specific approaches. Speaker-agnostic methods aim to learn universal forensic features applicable to any voice. Common architectures in this domain, such as RawNet2 [Tak et al., 2021b] and AASIST [Jung et al., 2022a], employ sinc-convolutions and graph attention networks, respectively, to model spectrotemporal artifacts. Despite their success on standard benchmarks (e.g., ASVspoof), these "black-box" models often struggle with generalization against unseen generation algorithms and lack interpretability.

Conversely, Speaker-Specific or POI detection reframes the task as a verification problem, leveraging reference data to protect specific identities. Early works [Jung et al., 2022b] integrate automatic speaker verification with anti-spoofing to detect personalized clones, while recent advancements utilize speaker profiles to detect speaker-specific impersonation. Notably, emerging research has begun to investigate leveraging phoneme-level features for enhancing detection performance [Zhang et al., 2025b], or performing phoneme-level analysis on POI [Salvi et al., 2025]. Our work builds upon this interpretable direction by establishing a comprehensive Chinese POI benchmark and introducing an adaptive profiling mechanism that explicitly models these idiosyncratic articulatory patterns.

3 Dataset Construction

To facilitate the study of POI deepfake detection under realistic conditions, we construct a large-scale speaker-centric Mandarin dataset containing both genuine and synthesized speech from public figures. The dataset is designed to benchmark personalized impersonation attacks, enabling finegrained analysis of speaker-dependent artifacts.

Data Collection and Preprocessing. We collect approximately 400 hours of real speech from 10 target speakers on major Chinese online streaming platforms. To ensure acoustic consistency, all recordings are processed using a voice activity detection (VAD) tool to extract speech-only segments. Non-speech regions and excessively short or long segments are discarded.

To further guarantee speaker purity, we leverage pretrained automatic speaker verification (ASV) embeddings to verify speaker identity. Segments whose embeddings deviate significantly from the target speaker profile are removed, effectively filtering out background speakers, interviewers, and cross-talk commonly present in in-the-wild recordings.

Speech Synthesis. For each target speaker, we generate spoofed speech using five representative zero-shot textto-speech (TTS) systems: F5-TTS, IndexTTS, LLaSA, OpenAudio-S1-mini, and VOXCPM. The selected TTS and VC systems are chosen to cover a wide range of modern zeroshot speech synthesis paradigms, including diffusion-based, autoregressive, and large language model (LLM) based architectures. All systems support reference-based voice cloning, enabling speaker-specific synthesis without fine-tuning. For each synthesis, a single genuine utterance from the target speaker is used as the reference signal, and the synthesis text is kept identical to the corresponding real speech.

Post-processing. All real and synthesized audio samples are resampled to a unified sampling rate and undergo identical post-processing procedures to eliminate confounding factors introduced by format or codec differences. The resulting dataset provides a clean and balanced benchmark for evaluating speaker-specific deepfake detection methods.

Dataset Statistics and Composition. Table 1 summarizes the overall statistics of the constructed dataset. The dataset covers ten target speakers with balanced real and synthesized utterances. For synthesized data, samples are evenly generated by multiple zero-shot TTS and VC systems to ensure diversity in generation mechanisms and artifacts.

4 Methodology

We propose a speaker-specific and phoneme-aware speech deepfake detection framework designed for POI scenarios. Given a small amount of reference speech from a target speaker, our method constructs a personalized phoneme-level acoustic profile and performs interpretable detection by measuring phonetic consistency at inference time.

4.1 Phoneme-aware Feature Representation

To achieve a comprehensive defense, we extract features at two distinct granularities: micro-level phoneme representations and macro-level speaker embeddings.

Phoneme Feature Extraction: We employ SSL model fine-tuned on speech deepfake datasets as the backbone F. For a given input x, the model extracts a sequence of highdimensional acoustic embeddings:

where T is the number of frames and D is the embedding dimension. These features encompass rich prosodic and phonetic information learned from diverse linguistic contexts.

Simultaneously, a Connectionist Temporal Classification (CTC) based aligner A is utilized to determine the phonetic boundaries. For each phoneme p appearing in the utterance, the aligner provides its temporal interval [tstart, tend]. To obtain a fixed-length representation for each phoneme instance, we apply a mean-pooling operation over the corresponding frame indices:

The resulting vector vp serves as a micro-level descriptor of the speaker's idiosyncratic pronunciation for phoneme p.

Global Speaker Embedding: To capture the holistic vocal identity, we utilize a pre-trained ASV model, denoted as Fspk. For the input utterance x, we extract a fixeddimensional global speaker embedding:

This embedding encapsulates long-term spectral and prosodic characteristics, providing a robust identity anchor complementary to the transient phonetic features.

4.2 Adaptive Speaker Profiling

A single phoneme may exhibit multiple articulation modes for the same speaker due to coarticulation, prosodic variation, and speaking style. To explicitly model such intra-speaker phonetic variability, we employ GMMs to model the distribution of each phoneme in the speaker's acoustic space. This formulation enables flexible modeling of intra-speaker variability while retaining a compact and interpretable statistical representation.

Phoneme Statistical Modeling: Let ES = {xi}Ni=1 denote the reference utterances of the target speaker. For each unique phoneme p in the reference set ES, we fit a GMM with Kp components. The probability density function for a phoneme vector v is defined as:

where λp = {πk, µk, Σk}Kp k=1 denotes the model parameters, including mixture weights πk, mean vectors µk, and covariance matrices Σk. To prevent over-fitting on sparse data, we restrict Σk to be a diagonal covariance matrix and adaptively adjust K based on the sample size Np of the phoneme.

Profile Reliability Weighting: Not all phonemes are equally discriminative or stable for a given speaker. To quantify the reliability of a phoneme profile, we compute a confidence weight wp based on the average log-likelihood ( ¯ Lp) of the reference samples:

The weight wp is then derived using an exponential scaling factor α:

A higher wp indicates that the speaker's pronunciation of phoneme p is highly consistent (compact in the feature space), making it a more reliable "fingerprint" for subsequent deepfake detection.

Global Identity Modeling: To model the speaker's holistic variance, we fit a separate GMM, denoted as λspk, to the set of global embeddings {ei}Ni=1 extracted from the reference utterances ES. Similar to the phonetic branch, we employ a diagonal covariance matrix to prevent overfitting on limited data:

where Kspk is number of mixture components. This global model serves as a coarse-grained verifier to ensure the test utterance falls within the target speaker's general acoustic manifold.

4.3 Identification of Salient Phoneme Fingerprints

Phonemes differ in their stability and speaker-discriminative capacity. Rather than treating all phonetic units equally, we identify a speaker-specific subset of salient phoneme fingerprints, denoted as Psalient, which capture the most consistent articulatory patterns of the POI.

Let Pall denote the set of phoneme types observed in the reference utterances of the target speaker. As described in Algorithm 1, the selection of salient phonemes is driven by the phoneme reliability weights wp introduced in Section 4.2. Specifically, phonemes in Pall are ranked according to wp, and only the top-K most reliable units are retained to form Psalient. By focusing on phonemes that exhibit high intraspeaker consistency, this selection yields a compact, speakerspecific, and interpretable phonetic fingerprint.

4.4 Tiered DecisionMechanism

In practical POI scenarios, test utterances are often short or linguistically constrained, leading to incomplete phoneme coverage. Let Ptest denote the set of phonemes detected in a test utterance. Relying solely on a fixed set of salient phonemes may therefore result in unreliable or missing decisions. To address this issue, we adopt a tiered decision mechanism that adaptively selects the most informative level of phonetic evidence available at inference time.

As illustrated in Algorithm 1, the decision process proceeds in a coarse-to-fine manner. If Ptest contains any speaker-specific salient phonemes, the final score is computed as a reliability-weighted average of their normalized phoneme-level scores sp, yielding a high-precision and interpretable decision. When no salient phonemes are observed Algorithm 1 Speaker-specific Salient Phoneme Modeling and Tiered Decision Input: Enrollment set Pall, weights {wp}, test utterance X, detected phonemes Ptest, parameter K.

Output: Detection score S(X).

// Salient Phoneme Fingerprint Construction

• 2: Psalient ← SelectTopK(Psorted,K) // Tiered Decision Mechanism
• 3: if Ptest ∩Psalient ̸= ∅ then
• 4: S(X) ←WeightedAvg ({sp}p∈Ptest∩Psalient , {wp})
• 5: else if Ptest ∩Pall ̸= ∅ then

6: S(X) ←Avg ({sp}p∈Ptest∩Pall )

• 7: else

8: Ctest ← {C(p) | p ∈ Ptest}

9: S(X) ←Avg ({sc}c∈Ctest )

• 10: end if

11: return S(X)

but other reference phonemes are present, the system falls back to a weighted average over all available phoneme-level scores. In the extreme case where phoneme overlap is sparse, phonemes are mapped to broader phonetic categories via a predefined function C(·) (e.g., vowels, plosives or fricatives), and category-level scores sc are used to ensure coverage.

This tiered approach prioritizes discriminative, speakerconsistent phonemes whenever possible, while maintaining robustness against linguistic sparsity and content mismatch.

4.5 Hybrid Scoring and Fusion Strategy

Before combining phoneme-level evidence, we normalize raw likelihood scores to ensure comparability across phonemes. For each phoneme p, we compute its loglikelihood Lp = logP(vp | λp) under the corresponding phoneme GMM, and apply a Sigmoid-based normalization:

where β and γ are the centering and scaling hyper-parameters determined empirically.

Finally, we fuse phoneme-level evidence with global speaker identity to produce the final detection score. We employ the tiered decision mechanism to compute the phonemelevel score Sphn(X). Parallelly, we compute the loglikelihood of the test utterance's global embedding etest given the global profile λspk, followed by Sigmoid normal-

where βspk and γspk are normalization parameters centered on the impostor distribution.

To balance the sensitivity to local artifacts and the robustness of global identity, the final detection score Sfinal is obtained via linear interpolation:

where α ∈ [0, 1] controls the trade-off between phonemelevel and speaker-level evidence.

5 Experiments

5.1 Implementation Details

Datasets. We evaluate our method on two POI deepfake datasets to assess its effectiveness and robustness under diverse linguistic and attack conditions. In addition to our proposed Chinese POI dataset, we adopt the Famous Figures dataset [Ali et al., 2025], an English POI deepfake benchmark featuring diverse speech synthesis attacks under zeroshot, few-shot, and fine-tuned settings. Both our dataset (denoted as ZH-Famous) and Famous Figures (denoted as ENFamous) are designed for POI scenario and include previously unseen synthesis algorithms, making them well suited for assessing speaker-centric and attack-agnostic defenses.

Model Configuration. For phoneme-level alignment, we employ a wav2vec2-large-xlsr-53 model 1 to extract phoneme boundary timesteps. For speaker-level representation, we adopt a pretrained ECAPA-TDNN model 2 to extract fixed-dimensional global speaker embeddings.

Each phoneme is modeled using a GMM with diagonal covariance matrices. Unless otherwise specified, we utilize 1% of the bona fide speech from each speaker's available data as reference data for all experiments. We fix the number of mixture components to Kp = 5 for phoneme-level models and Kspk = 5 for the global speaker model, together with a covariance regularization term of 10-3 for numerical stability. The number of salient phonemes is set to K = 12 across all experiments. For likelihood normalization, we apply a sigmoid mapping with β = -2000 and γ = 200. For final score fusion, we set α = 0.8.

Evaluation Metrics. We report Equal Error Rate (EER, %) and Area Under the ROC Curve (AUC, %) as the primary evaluation metrics. Lower EER and higher AUC indicate better detection performance.

5.2 Plug-and-Play Evaluation

A core design goal of the proposed Phoneme-based Voice Profiling (PVP) module is plug-and-play compatibility with existing deepfake detection backbones. To validate this property, we integrate PVP with a diverse set of SSL-based encoders [Ge et al., 2025] and evaluate performance on POI and cross-lingual benchmarks.

1 https://huggingface.co/facebook/wav2vec2-xlsr-53-espeak-cv-ft

2 https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb

As summarized in Table 2, incorporating PVP consistently improves detection performance across all backbones on both datasets. On the EN-Famous benchmark, PVP integration yields an average EER reduction of approximately 8.3% and an average AUC improvement of about 5.6% across different backbones. On the ZH-Famous dataset, PVP brings more substantial gains, achieving an average EER reduction of around 12.1% and an average AUC increase of approximately 13.6%, demonstrating strong effectiveness under cross-lingual and unseen generation conditions.

5.3 Comparison with State-of-the-Art Methods

We compare our method with state-of-the-art deepfake detection approaches under a unified speaker-centric evaluation protocol. As shown in Table 3, the proposed method consistently achieves the best performance across both benchmarks, with substantially higher AUC and lower EER than all competing approaches.

Compared to general-purpose detectors, our framework exhibits a clear advantage in capturing fine-grained, speakerdependent articulatory patterns that are critical for detecting POI speech synthesis. In particular, the large performance margins indicate that modeling phoneme-level consistency from bona fide speech provides strong robustness against unseen spoofing attacks, even without exposure to attack-specific data during modeling. Despite being designed for personalized defense, the method also maintains strong performance across both English and Chinese benchmarks, demonstrating favorable cross-lingual generalization.

5.4 Ablation Study

We conduct ablation studies to examine the impact of key design choices in the proposed PVP framework under the same POI evaluation protocol. Specifically, we replace phonemelevel GMM likelihood modeling with cosine similarity between phoneme embeddings, remove the global speaker embedding branch, and discard phoneme-level profiling in favor of utterance-level modeling, respectively.

Table 4 shows that each component of PVP contributes positively to detection performance under the POI benchmarks. Removing phoneme-level profiling and reverting to utterance-level modeling causes the most consistent degradation, confirming the importance of fine-grained phonetic analysis for capturing speaker-specific articulatory regularities. While certain components may exhibit slightly better performance under specific conditions, their effectiveness is not consistently preserved across datasets and metrics. In contrast, the full PVP framework, which integrates phoneme-level modeling, probabilistic characterization, and global speaker cues, delivers the most stable and averagedbest performance. These results suggest that the strength of PVP lies not in any single module, but in the complementary interplay among its components.

5.5 Interpretability and Case Study

Beyond detection accuracy, our proposed PVP framework provides inherent interpretability by explicitly decomposing the final decision into phoneme-level, speaker-conditioned evidence. Unlike conventional end-to-end detectors that output utterance-level black-box scores, PVP allows each phonetic unit to be independently evaluated against the POI's articulatory profile learned from bona fide reference speech, making the detection process transparent and traceable.

This focus on phoneme-level speaker inconsistency provides actionable forensic cues while preserving robustness and generalization to previously unseen attacks. Such interpretability arises naturally from the design of PVP rather than post-hoc analysis: by enforcing phoneme-level alignment and probabilistic modeling, detection decisions are grounded in deviations from a speaker's own phonetic distribution instead of attack-specific patterns. As illustrated in Fig 4, bona fide speech maintains consistently high confidence across segments, whereas the synthetic clone exhibits localized "red flags" at specific phonemes where the generative model fails to replicate the POI's unique articulatory habits. Moreover, such structured, phoneme-level evidence offers a promising interface for future reasoning-based or large-model-driven forensic systems, where explicit articulatory consistency can serve as interpretable intermediate signals.

6 Conclusion

This paper introduces Phoneme-based Voice Profiling (PVP), a plug-and-play and lightweight framework for POI speech deepfake detection. By shifting detection from black-box utterance-level classification to phonetic consistency verification, PVP models speaker-specific articulatory patterns as personalized "fingerprints" using lightweight GMM-based statistics. This design enables high data efficiency and strong generalization to unseen synthesis algorithms, without relying on heavy spoof-specific training.

Extensive evaluations on both our newly curated ZHFamous dataset and Famous Figures dataset demonstrate that PVP consistently outperforms state-of-the-art methods across diverse SSL backbones, achieving substantial reductions in EER. Beyond detection performance, the proposed framework provides transparent phonetic anomaly cues by exposing speaker-centric inconsistencies at the phoneme level, thereby bridging automated deepfake detection with interpretable POI forensic analysis. These findings suggest that phoneme-level voice profiling offers a principled and extensible direction for personalized and interpretable defense against POI speech synthesis.

Acknowledgements

This work is supported by the Natural Science Foundation of China (NSFC) under the grant NO.62572358, 62372334