Breaking the Trade-off: Orthogonal Semantic Decoupling for Generalizable and Fair Deepfake Detection

1 Introduction

Recently, the rapid advances in generative AI techniques have significantly reduced the cost of synthesizing highly indistinguishable images [Rosberg et al., 2023; Wu and De La Torre, 2023; Kim et al., 2025]. Facial deepfakes, which focus on manipulating personal expressions, attributes, and identity, have severely threatened individual privacy and reputation as well as public safety [Wang et al., 2024]. Consequently, developing reliable and robust face deepfake detection methods has become an urgent need.

∗Corresponding author.

Early deepfake detectors primarily relied on manipulationspecific artifacts [Li and Lyu, 2019; Li et al., 2020a; Qian et al., 2020]. Such approaches suffer substantial performance drops on unseen manipulations [Luo et al., 2021; Xia et al., 2026]. To improve detector generalization, prior works have explored learning manipulation-invariant features [Liu et al., 2021; Tan et al., 2023], augmenting training data with generalized forgery traces [Shiohara and Yamasaki, 2022; Chen et al., 2022], and leveraging pretrained visual foundation models [Radford et al., 2021; Ojha et al., 2023; Wu et al., 2025; Yan et al., 2025]. Besides, given that faces are demographically correlated, detector performance appears biased across different demographic groups, as shown empirically in [Lin et al., 2025]. To improve detection fairness, recent studies have explored feature disentanglement [Lin et al., 2024], riskbased group-aware optimization [Ju et al., 2024], and causal interventions [Cheng et al., 2025b] to mitigate demographic bias and reduce performance disparities across subgroups.

However, prior works typically target either generalization or fairness in isolation. Generalization-oriented optimization is often dominated by the gradients of the majority-group population [Lin et al., 2024], thereby biasing models toward majority-favored cues (Figure 1(a)). Fairness-oriented constraints can reduce subgroup disparities to improve detection performance under minority-group populations. Such enforced decision-boundary alignment, however, impacts discriminability and ultimately decreases cross-domain generalization [Cheng et al., 2025b] (Figure 1(b)). As a result, it seems that there may be a trade-off between generalization and fairness, which impacts real-world detector deployments.

To improve both generalization and fairness, we revisit a structural perspective on where forgeries come from. Since facial deepfake images are always manipulated from real faces, there are high semantic similarities within the extracted feature space. The determination of authenticity can be restricted to a smaller, semantically consistent subspace.

Based on this insight, we propose Orthogonal Semantic Decoupling (OSD), a framework that constructs an orthogonal, low-rank mixture-of-experts in the residual subspace of a vision-language model (VLM). Specifically, OSD applies Singular Value Decomposition to the pretrained weights of the VLM to orthogonally construct the principal subspace and the residual subspace. Within the residual subspace, OSD instantiates a set of mutually orthogonal Demographic Semantic Experts (DSEs), which are selected through a text-guided and training-free router to adapt to demographic-specific distribution shifts. In parallel, a Universal Forgery Expert (UFE) is introduced to capture transferable face forgery cues shared across domains and demographics. By factorizing the decision space into mutually orthogonal demographic subgroup subspaces, OSD suppresses cross-domain interference and demographic-induced spurious correlations, while enabling a clear within-subgroup distinction in a compact subspace (Figure 1(c)), thereby effectively breaking the trade-off.

The contributions of this paper are summarized as follows:

• We propose Orthogonal Semantic Decoupling (OSD), a framework for generalizable and fair deepfake detection. OSD operates in the residual visual feature subspace and explicitly decouples Demographic Semantic Experts (DSEs) and a Universal Forgery Expert (UFE), which suppresses demographic-forgery correlations.
• We develop a training strategy for OSD that encourages the model to focus on demographic-specific and demographic-agnostic forgery cues, respectively. Based on the image-text alignment in VLM, DSEs are accurately selected with a text-guided and training-free router during inference.
• Extensive experiments on multiple face deepfake benchmarks demonstrate improvements in detection generalization as well as fairness compared with state-of-theart methods, effectively breaking the generalizationfairness trade-off.

2 RelatedWork

2.1 Generalization in Deepfake Detection

Improving cross-domain generalization under unseen manipulations remains an active research focus. Existing generalization-oriented methods broadly fall into two categories: forgery-pattern learning and adaptation of pretrained visual foundation models.

Forgery pattern learning improves transferability by mining low-level artifacts or statistical inconsistencies shared across manipulation methods. F3-NET [Qian et al., 2020] and SPSL [Liu et al., 2021] model spectral anomalies in the frequency domain, while LGRAD [Tan et al., 2023] learns more generalizable forgery representations via gradientbased responses. In the spatial domain, FACE X-RAY [Li et al., 2020a] and IID [Huang et al., 2023] exploit blendingboundary cues and identity-consistency constraints, respectively. In addition, reconstruction-based anomaly detection methods, such as RECCE [Cao et al., 2022] and DIRE [Wang et al., 2023], increase sensitivity to unseen generators using reconstruction errors or inversion-related features. To reduce reliance on a specific training distribution, SBI [Shiohara and Yamasaki, 2022] and SLADD [Chen et al., 2022] further introduce self-supervised or self-synthesized strategies to increase the diversity of forged samples.

Adaptations of visual foundation models have emerged as an important trend, leveraging the generalizable representations provided by visual foundation models to improve generalization. UNIFD [Ojha et al., 2023] performs classification in a fixed, general-purpose feature space, reducing overfitting to specific generator distributions. LASTED [Wu et al., 2025] introduces semantic-level priors into detector training, for example via language-guided contrastive learning. EFFORT [Yan et al., 2025] shows that naive fine-tuning may cause feature-space collapse and proposes an SVD-based orthogonal subspace decomposition: it freezes the principal components to preserve pretrained semantics and learns forgery patterns only in the residual subspace.

2.2 Fairness in Deepfake Detection

As deepfake detection continues to advance, performance disparities across demographic groups, such as race and gender, have received growing attention. Trinh et al. [Trinh and Liu, 2021] argue that imbalanced training distributions are a primary driver of group disparities and can encourage detectors to rely on spurious correlations. Building on this line of work, Xu et al. [Xu et al., 2024] construct large-scale, fine-grained attribute annotations and a slice-based evaluation protocol, enabling more precise measurement of demographic bias and potential failure risks. To mitigate these issues, recent studies aim to dynamically reduce the influence of sensitive attributes during representation learning or decision making. Lin et al. [Lin et al., 2024] propose the PFG framework, which uses decoupled learning to extract demographic-invariant forgery features. From a risk-optimization perspective, Ju et al. [Ju et al., 2024] introduce a CVaR-based fairness objective to improve robustness across groups by targeting the worst-case group or samples. Cheng et al. [Cheng et al., 2025b] connect fairness to cross-domain generalization via causal analysis and mitigate demographic bias through demographic-aware rebalancing and demographic-agnostic feature aggregation.

3 Methodology

3.1 ProblemDefinition

Face deepfake detection is formulated as a binary classification problem: given a face image x, a detector outputs ˆ y ∈ {0, 1}, where ˆ y = 0 denotes a real face and ˆ y = 1 denotes a synthesized (forged) face. In practical deployment, we consider two objectives for face deepfake detection: crossdomain generalization and demographic fairness. Let a ∈ A denote a sensitive attribute (e.g., race or gender). We quantify generalization by a metric G(·) and fairness by a metric F(·) that captures performance disparities across subpopulations. A desirable detector should perform well on both objectives, remaining robust under distribution shifts while behaving consistently across demographic groups. However, prior works typically optimize these objectives in isolation, i.e., improving either G or F while overlooking the other, as illustrated in Figure 2. Therefore, our goal is to develop a detector that simultaneously improves generalization and fairness, achieving strong performance on both G and F.

3.2 Overview of the Proposed Framework

To improve both fairness and cross-domain generalization in face deepfake detection, we propose Orthogonal Semantic Decoupling (OSD). Our key observation is that demographic attributes, such as race and gender, define relatively stable high-level semantics: they shift the facial appearance distribution but are unrelated to whether an image is forged, making them a natural basis for forgery-detection subspace partitioning. Based on this, we perform deepfake detection within smaller, semantically consistent, and mutually independent subspaces. Figure 3 provides an overview of OSD.

OSD operationalizes this idea on top of a pretrained vision-language model (VLM) by freezing a shared principal subspace for general representations and learning only a set of low-rank residual experts. Concretely, we instantiate Demographic Semantic Experts indexed by race × gender, and route each input to the corresponding expert based on the similarity between image embeddings and text embeddings of predefined demographic descriptions; in parallel, a Universal Forgery Expert is always activated to extract demographic-invariant manipulation cues. The final prediction combines expert-specific forgery cues from the routed Demographic Semantic Expert with universal forgery cues from the Universal Forgery Expert. To encourage complementary and separable representations, we enforce pairwise orthogonality among all Demographic Semantic Experts as well as orthogonality between each expert and the frozen principal subspace. We adopt a two-stage training procedure that learns demographic-specific and demographic-agnostic forgery cues separately, improving both cross-group consistency and cross-domain generalization.

3.3 Demographic-Conditioned Residual Experts

Let W ∈ Rdo×di denote the pretrained weight of a selfattention projection in a vision-language model (VLM) (e.g., one of the Q/K/V or output projection matrices). We compute the SVD W = UΣV ⊤ and keep the top rmain singular components (with corresponding Ur, Σr, Vr) to form a frozen principal subspace

We define the residual component as Wres = W -Wmain. Both the Demographic Semantic Experts and the Universal Forgery Expert are initialized from this residual subspace by using the leading singular directions of Wres to initialize each expert's low-rank factors. On top of this, each expert e learns a low-rank residual update

where Ue ∈ Rdo×rexp , Ve ∈ Rdi×rexp , se ∈ Rrexp , and rexp ≪ min(do, di). Given an input x, we obtain its image embedding zI = fI(x) and select the expert index e∗ (x) using a text-guided router (see Sec. 3.4 for construction details). Given an input feature h ∈ Rdi , the layer output is

Here, f denotes the Universal Forgery Expert, and λs and λf are constant fusion weights for the routed Demographic Semantic Expert and the Universal Forgery Expert, respectively.

To prevent different experts from updating the model along the same directions and interfering with each other, we impose an orthogonality regularization inspired by OMNIAID [Guo et al., 2025] on the subspace spanned by the lowrank residual updates. Concretely, we encourage each active expert's bases (Ue, Ve) to be orthogonal to the frozen principal subspace (Ur, Vr) and, for DSEs, to the previously trained demographic experts. We define the orthogonality loss as

where ¯U and ¯ V denote column-normalized bases, and M denotes the frozen principal subspace. We set Cei = {M, e1, . . . , ei-1} for the i-th DSE and Cf = {M} for the UFE, and average the loss over all SVD-MoE projections.

3.4 Specialization to Balanced Integration

To effectively decouple demographic semantics from forgery artifacts and avoid expert entanglement under end-to-end joint optimization, OSD adopts a two-stage training strategy that progresses from specialization to integration. The core idea is to first strengthen the independent modeling capability of different types of experts, and then perform holistic collaboration under a fixed decoupled structure.

Expert Specialization. In the first stage, we train the Demographic Semantic Experts (DSEs) and the Universal Forgery Expert (UFE) separately to establish clear functional roles. For DSEs, we apply group-aware supervision with group sampling: for each sample, only the corresponding demographic expert is updated, allowing appearance and texture variations related to race and gender to be absorbed by expert specialization rather than interfering with forgery modeling. Additionally, we adopt class-balanced sampling to alleviate within-group real/fake imbalance during DSE training.

To encourage the UFE to learn transferable forgery cues rather than manipulation-specific artifacts, we train it on all real images in the training set, together with synthetic forged samples generated from these real faces via SelfReconstruction-Blended Images (SRBI). Specifically, we extract facial boundaries and landmarks, obtain latent-space reconstructed frames using a pretrained autoencoder, and then randomly select the target from either the original frame or the reconstructed frame. Pixel-level blending with dynamic masks is applied to introduce more diverse statistical discontinuities and forgery patterns, thereby encouraging the UFE to learn more general artifact representations.

End-to-End Tuning. To promote collaborative inference between the DSEs and the UFE, OSD enters an integration stage to stabilize how expert outputs are combined for final decisions. Accordingly, we freeze the principal subspace as well as all experts' parameters, and fine-tune only the classification head with cross-entropy loss. To mitigate imbalance across demographic subgroups, we use demographicweighted sampling when fine-tuning the head, where each training sample is weighted by the inverse of its subgroup frequency. Meanwhile, expert routing is handled by a trainingfree, text-guided router that leverages image-text similarity in the pretrained VLM to activate the appropriate DSE. Specifically, we predefine a set of demographic descriptions {τe}e∈E (e.g., race × gender) and compute their text embeddings {zeT }e∈E, where each zeT is obtained by encoding a set of prompt templates and averaging the normalized text features into a prototype. For an input image x, we extract its image embedding zI and select the expert via similarity matching:

where sim(·, ·) denotes cosine similarity between ℓ2normalized embeddings (optionally scaled by a temperature τ). The training procedure is summarized in Algorithm 1.

Algorithm 1 Two-stage training for OSD

Input: Training set Dtrain = {(x, y, a)} Params: Demographic experts {fe}Ee=1 , forgery expert f, classifier head g Hyper: λorth (orth. weight); Tsem, Tf , Tg (iters per stage) Output: Trained {fe}Ee=1 , f, g 1: Stage 1: Specialization 2: for e = 1 to E do 3: for t = 1 to Tsem do 4: B ∼ GroupSampling(Dtrain | a = e) 5: L←Ldet(B;Wmain, fe, g) +λorthL(e) orth 6: Update({fe, g}, L) 7: end for 8: end for 9: for t = 1 to Tf do 10: B ∼ (Dtrain | y = 0) 11: ˜ B = SRBI(B) 12: L←Ldet(B ∪ ˜ B; Wmain, f, g) + λorthL(f ) orth 13: Update({f, g}, L) 14: end for 15: Stage 2: Integration 16: Freeze Wmain, all fe, and f, and fine-tune g. 17: for t = 1 to Tg do 18: B ∼ Dtrain 19: for (x, y, a) ∈ B do 20: e∗ (x) ← Route(x) using Eq. 5 21: end for 22: L←Ldet(B;Wmain, fe∗ (·), f, g) 23: Update(g, L) 24: end for

4 Experiments

4.1 Experimental Setup

Datasets. Following prior work [Yan et al., 2023; Yan et al., 2025], we use FaceForensics++ (FF++, c23) [R¨ossler et al., 2019] as the training dataset, and evaluate our model on multiple public face deepfake detection benchmarks, including DFDC [Dolhansky et al., 2020], DFD [Dufour et al., 2019], and CDFv2 [Li et al., 2020b], to assess crossdomain generalization performance. To further evaluate fairness across demographic subgroups, we follow the annotation and cross-group partitioning strategy adopted in recent fairness-oriented deepfake detection studies [Ju et al., 2024; Lin et al., 2024; Xu et al., 2024]. Specifically, following prior fairness-oriented studies, we adopt their gender and race annotations and cross-group partitioning, forming six intersectional subgroups by combining these attributes: Asian-Male (A-M), Asian-Female (A-F), White-Male (W-M), WhiteFemale (W-F), Black-Male (B-M), and Black-Female (B-F).

Evaluation Metrics. We use the frame-level Area Under the ROC Curve (AUC) as the primary metric to evaluate model generalization performance. For fairness, we report Skew [Geyik et al., 2019; Wang and Deng, 2020; Cheng et al., 2025a], which measures the maximum performance disparity across demographic subgroups; lower values indicate smaller disparities and better fairness.

Implementation Details. We adopt CLIP ViT-L/14 as the default vision-language model (VLM) backbone. For our method, all input frames are resized to 224 × 224. The expert fusion weights are set to λs = 0.5 and λf = 0.5, the expert rank to rexp = 8, and the orthogonality loss weight to λorth = 0.001 by default. All experiments are conducted on a single NVIDIA H20 GPU for both training and evaluation.

4.2 Comparison Experiments

To evaluate cross-domain generalization and fairness, we compare OSD with representative deepfake detectors from two lines of research: (i) generalization-oriented methods (e.g., Xception [R¨ossler et al., 2019], EfficientNet [Tan and Le, 2019], F3 -Net [Qian et al., 2020], Face X-ray [Li et al., 2020a], SBI [Shiohara and Yamasaki, 2022], RECCE [Cao et al., 2022], GRU [Choi et al., 2024], CADDM [Dong et al., 2023], UCF [Yan et al., 2023], ProDet [Cheng et al., 2024], VLFFD [Sun et al., 2025], and EFFORT [Yan et al., 2025]) and (ii) fairness-oriented methods (e.g., DAW-FDD [Ju et al., 2024], FG [Lin et al., 2024], and DAID [Cheng et al., 2025b]). All methods are trained on FF++ (c23) and evaluated on DFDC, DFD, and CDFv2. We report frame-level AUC for generalization and Skew across six intersectional demographic groups (race×gender) for fairness. The results are summarized in Table 1.

As shown in Table 1, OSD achieves state-of-the-art performance in both cross-domain generalization and demographic fairness under the same cross-dataset protocol. In terms of generalization, OSD attains the best Avg AUC (84.94%) across DFDC, DFD, and CDFv2, with EFFORT being the closest generalization-oriented competitor. In terms of fairness, OSD achieves the lowest Avg Skew (0.659). We also observe that Skew values on DFDC are consistently higher across methods, likely because DFDC provides the most complete and relatively balanced intersectional demographic coverage; therefore, we select DFDC for the visualization analyses in Sec. 4.4.

4.3 Ablation Experiments

We conduct ablation studies to quantify the contribution of each component in OSD. Unless otherwise specified, all ablations follow the same training protocol as in Sec. 4.1 and are evaluated under the same cross-domain setting as in Sec. 4.2.

Comparison onModules. We ablate the three core components of OSD: Demographic Semantic Experts (DSEs), Universal Forgery Expert (UFE), and text-guided routing (R). As shown in Table 2, enabling UFE alone improves AUC (59.1%→65.0%) but increases Skew (1.50→1.61), indicating that forgery-only modeling without demographic decoupling leads to disparate generalization across subgroups. Enabling DSEs alone reduces Skew (1.50→1.17) with moderate AUC gains, suggesting that subgroup-specific adaptation effectively balances cross-group performance. Combining both yields substantial improvements in both metrics, and adding text-guided routing further boosts performance, demonstrating that proper expert selection enhances the synergy between the two branches. To verify that the gains are not simply from SRBI, we adapt CLIP with LoRA using the same SRBI. Its 74.5% Avg AUC remains much lower than OSD's 84.9%.

Comparison onHyperparameters. We examine the sensitivity to the orthogonality loss weight λorth and the principal rank rmain in Table 3. Removing the orthogonality constraint (λorth = 0) leads to higher Skew, indicating that expert representations become entangled without explicit decoupling. Overly aggressive constraints degrade AUC due to limited expressiveness. For the principal rank, rmain controls howmuch pretrained semantics are kept in the frozen principal subspace versus delegated to the residual experts. Too small principal ranks (e.g., 1024-16) under-preserve pretrained semantics, while too large principal ranks (e.g., 1024-4) leave limited residual capacity for adaptation; a moderate rank (1024-8) achieves the best balance.

We further analyze the effect of expert fusion weights λs and λf in Figure 4. To isolate the influence of each weight, we fix one at 0.5 while varying the other. Since the principal subspace component Wmainh is frozen and always included, overly small weights (especially when both λs and λf are small) make the prediction dominated by Wmainh, resulting in limited task-specific adaptation and poor AUC. Increasing λs or λf allows the routed DSE or the UFE to contribute effective residual updates, improving performance. However, overly large weights can make the residual adaptation dominated by a single branch, weakening the complementary benefits of jointly leveraging DSE and UFE and thus hurting generalization on CDFv2.

4.4 Visualization

Feature-Space Decoupling. To validate the effect of orthogonal semantic decoupling, we visualize the learned feature representations via t-SNE. We select DFDC for visualization as it is the only test set that covers all six intersectional demographic subgroups with relatively balanced sample counts and both real and fake samples in each group, enabling a complete and fair view of demographic decoupling. We compare OSD with EFFORT [Yan et al., 2025], the most related baseline that also employs SVD-based orthogonal subspace adaptation on vision-language models. While EFFORT focuses solely on learning forgery patterns in the residual subspace, it does not explicitly decouple demographic semantics from forgery cues. This comparison isolates the contribution of our demographic-aware expert design.

As shown in Figure 5, each point represents a test sample; colors indicate six intersectional demographic groups: Asian-Male (A-M), Asian-Female (A-F), White-Male (WM), White-Female (W-F), Black-Male (B-M), and BlackFemale (B-F); lighter points denote fake samples and darker points denote real samples. In EFFORT (left), samples from different groups are substantially entangled, and the real/fake separation varies across groups, suggesting reliance on demographic shortcuts and cross-subgroup interference. In contrast, OSD (right) yields clearer group-wise clusters with more consistent real/fake boundaries within each subgroup, indicating that orthogonal decoupling separates demographic semantics from forgery cues and stabilizes decision boundaries across groups, improving fairness and robustness.

Attention Heatmaps. To verify that OSD focuses on forgery artifacts rather than demographic appearance, we visualize attention heatmaps on forged samples from the six demographic subgroups in DFDC. As shown in Figure 6, EFFORT (top row) tends to attend to facial morphology and facial-feature structures that correlate with demographic attributes, and its attended regions vary noticeably across subgroups. In contrast, OSD (bottom row) consistently highlights forgery-related regions such as blending boundaries and texture inconsistencies across subgroups. This cross-group consistency indicates that orthogonal decoupling effectively suppresses demographic shortcuts, enabling demographic-agnostic forgery detection.

5 Conclusion

This paper analyzes the practical trade-off between crossdomain generalization and demographic fairness in face deepfake detection. We propose Orthogonal Semantic Decoupling (OSD), which applies SVD to a pretrained visionlanguage model (VLM) to separate a principal semantic subspace and a residual subspace, and learns an orthogonal mixture of experts in the residual space to suppress demographic shortcuts and cross-subgroup interference. Extensive experiments on DFDC, DFD, and CDFv2 show that OSD achieves the best Avg AUC (84.94%) and the lowest Avg Skew (0.659), achieving state-of-the-art performance in both generalization and fairness. In real-world deployment, the text-guided, training-free router may be less reliable when demographic attributes are ambiguous or more fine-grained, and may be slightly affected by inherent biases in the VLM. Future work can explore continual residual-expert adaptation to new domains or attributes and extend the router to finegrained demographic settings by using a larger set of tunable demographic prompts to improve image-text alignment.

Contribution Statement

Zhongyu Shi and Siyu Peng contributed equally to this paper.