Robust, Generalizable Proactive Face-swapping Defense via Semantic Gradient Divergence

1 Introduction

Face-swapping technology (a subset of deepfakes) substitutes the facial identity of a subject in visual media with that of another. Recent progress in identity-feature-based methods has enabled the generation of swaps for unseen target images without requiring supplementary data or retraining. Although these innovations facilitate legitimate uses in industry (e.g., entertainment, gaming, and digital avatar design), they pose significant ethical risks, including identity theft, the spread of disinformation, and reputational damage.

∗ Corresponding author

Recent research has focused on two paradigms for risk mitigation: deepfake detection and proactive defense. Although detection methods [Tao et al., 2025] identify manipulated content post hoc and cannot curb its spread, proactive defense protects users by disrupting identity feature extraction at the source. However, most existing proactive approaches [Komkov and Petiushko, 2021; Sun et al., 2024; Jeong et al., 2025] are tailored to a single deepfake backbone and thus generalize poorly across face-swapping models (Fig. 1(a)). Moreover, in pursuing stronger protection without patches or explicit noise optimization, they often introduce perceptible artifacts that degrade image fidelity (Fig. 1(b)). Even defenses with less perceptible perturbations can be weakened by post-processing such as diffusion purification, JPEG compression, and transformations (Fig. 1(c)), leaving deepfakes perceptually close to the source identity and facial privacy vulnerable to exploitation.

This paper proposes a proactive face-swapping defense framework capable of generating natural-looking protected images that are robust against adaptive attacks and input transformations to address these limitations while demonstrating generalized performance across diverse deepfake generation backbones. The framework integrates key strategies to achieve these goals. First, it computes gradients on diffusion-purified images to iteratively purify adversarial noise during updates, constraining perturbations to remain imperceptible on the natural image manifold while resisting adaptive diffusion-based purification. Next, a feature gallery is created for the protection mechanism. Applying representative backbones of deepfake generation models as surrogate models, we semantically refine the extracted heterogeneous identity cues into a homogeneous identity feature.

This work proposes a proactive defense method that safeguards facial images by introducing distortions in deepfake generation output using a feature gallery of integrated features. The objective is to update perturbations that allow rapid distortion by identifying vector directions corresponding to crucial facial components in the entangled identity space. This work proposes a novel joint embedding space by semantically combining CLIP features and identity features. In this space, the representation is steered by consensus weighting toward the direction that induces the strongest distortion away from the protected image's original attributes. Based on these objectives, the resulting deepfakes undergo significant distortion to the extent that the source cannot be inferred. This work analyzes the feature shift in the identity space caused by applying these transformations to the purified image to ensure robustness against post-processing operations. Optimizing perturbations to counteract this shift and restoring features to their intended distorted state achieves strong resilience against purification attacks. The SGD-Guard approach demonstrates exceptional robustness against adaptive attacks by providing an effective defense with minimal visual artifacts while achieving efficient image generation with low computational latency. Our source code and appendix are available at https://github.com/BACKAI/SGD-Guard. The following are the principal contributions of this work:

• This work proposes a semantic distortion (SD) attack in a CLIP-identity joint embedding space (CI-JES), leveraging consensus weighting to target facial attributes and disrupting deepfake generation while preserving visual fidelity. By aligning high-level semantics with identity representations, the distortions transfer across domains.
• This work presents semantic direction expectation over transformation (EOT), replacing the standard uniform expectation with direction-adaptive weighting based on semantic shifts in the CI-JES. It prioritizes critical transformations and prevents gradient dilution for efficient and robust optimization.
• This work proposes a semantic iterative identity refinement (SIIR) via channelwise disagreement to refine heterogeneous identity embeddings into a homogeneous representation. It thereby produces an identity feature that generalizes across diverse deepfake generation backbones.

2 RelatedWork

2.1 Face-Swapping Deepfake

Face swapping transfers the identity of a source face onto a target image while preserving the target attributes (e.g., the expression, illumination, and pose). Early learning-based methods [Korshunova et al., 2017] often required subjectspecific training for each identity pair, limiting generalization to unseen identities. Recent subject-agnostic frameworks have enabled efficient identity transfer without finetuning. Regardless of the generative architecture, whether based on convolutional neural network encoder-decoder models [Wang et al., 2021], generators based on the generative adversarial network [Chen et al., 2020; Rosberg et al., 2023], or emerging diffusion models [Zhao et al., 2023; Kim et al., 2025; Baliah et al., 2025], the core mechanism includes feature injection and fusion. Source embeddings are inserted into the latent feature maps of the target, where a fusion module combines source identity cues with the structural attributes of the target. This work aims to disrupt identity transfers by perturbing the source embeddings before fusion, inducing semantic divergence in the synthesized output.

2.2 Proactive Face-Swapping Defense

Proactive face-swapping defense methods modify facial images so that edits become null or visually odd. A line of work relies on adversarial watermarking, where small, structured perturbations are embedded into facial images to interfere with downstream generators. Cross-model processes [Huang et al., 2022] and watermark-embedding networks [Zhang et al., 2024] follow this paradigm but typically assume whitebox access, limiting scalability.

Universal defenses have been proposed to relax this assumption, including transferable adversarial generators [Dong et al., 2023] and compression-aware perturbations [Wang et al., 2022; Qu et al., 2024]. However, these methods target reconstruction-based encoder-decoder processes and offer limited protection against identity-driven face swapping in an identity-embedding space.

Other approaches disrupt identity extraction by attacking face-recognition components [Hu et al., 2024] or applying local modifications [Ma et al., 2023], attribute editing [Jia et al., 2022], and makeup-based perturbations [Hu et al., 2022; Sun et al., 2024]. Although effective, these methods often require stronger perturbations or visible changes for robustness, degrading visual fidelity.

Recent methods mitigate such artifacts [Wang et al., 2025; Jeong et al., 2025] but may remain vulnerable to post-processing or depend on specific identity extractors.

This work proposes an adversarial watermarking framework driven by semantic gradient divergence to address these limitations, offering transferable protection robust to complex post-processing while preserving high visual fidelity.

3 ProposedMethod

3.1 Overview

This work proposes a proactive face-swapping defense that facilitates high visual fidelity and generalized robustness across diverse deepfake backbones. In Figure 2, the framework iteratively optimizes the perturbation δ via a diffusion-guided strategy to resist diffusion-based adaptive attacks [Xue et al., 2023]. Integrating one-step diffusion purification [Lei et al., 2025] imperceptible yet resilient noise while accelerating the optimization process. Prior to the perturbation update, unified identity features are synthesized from backbone outputs and are integrated with CLIP to create a stable reference gallery. This work applies gallery features and four identity prompts to estimate directions that maximize SD in deepfake outputs. To ensure robustness, this approach estimates transformation-induced latent shifts offline across the dataset and neutralizes them during optimization to prevent post-processing degradation.

3.2 Semantic Distortion Attack

Proactive deepfake defenses often fail to protect synthesized outputs because key identity-relevant facial attributes [Lestriandoko et al., 2022; Lee et al., 2023] remain intact, offering limited protection. Thus, this work assigns larger weights to distortion directions that strongly degrade identity-critical attributes, making the original image unidentifiable. Accordingly, we select four local identity-critical attributes (eyebrows, eyes, nose, and lips) as manipulation anchors, since broader cues such as pose or style are less identity-specific and may dilute the semantic distortion gradient. Further discussion and experiments are provided in Appendix A. We then construct the prompt set tj 4j=1 , where each tj denotes the CLIP text feature for the jth attribute. Figure 3 presents a cosine-similarity assessment between these prompts and the CLIP image features from a prebuilt feature gallery, yielding a set of similarity scores S:

where Ci denotes the CLIP image feature corresponding to the ith sample in the feature gallery.

From S, this method ranks the gallery samples for each attribute j and selects the top-k and bottom-k subsets.

To retrieve samples efficiently from S, this work uses FAISS [Douze et al., 2024] and averages the CLIP features in each subset to obtain Ctop and Cbot, and retrieves identity features from the same samples to calculate Itop and Ibot. This work concatenates the corresponding CLIP and identity centroids to form the joint anchors Atop and Abot for each attribute, producing four anchor pairs. Integrating features extracted from these distinct latent spaces creates CI-JES that enables the concurrent exploitation of semantic and identityrelated representations. Unlike isolated-space defenses, the framework synergizes high-level semantics and identity embeddings.

Next, this work computes the joint representation of the purified sample x∗ and encodes x∗ using a CLIP encoder Eclip and an identity encoder Eid to obtain Cx∗ and Ix∗ , which are combined to form Vjoint. The relative alignment toward the top and bottom anchors for each attribute is calculated to determine the distortion direction for Vjoint, which is initiated by defining vectors between the features of x∗ and each anchor in the CLIP and identity spaces:

Because CLIP and identity embeddings reside in heterogeneous spaces, we first map their directional vectors into a shared comparison space using lightweight projection heads, followed by ℓ2 normalization. Next, the cosine agreement for the top and bottom pairs in each space is computed as follows:

Then, the weight r for each anchor is calculated based on the alignment degree of the top and bottom pairs:

The consensus weight r assigns relative importance to the distortion direction of Vjoint. Calculating the similarity between Vjoint and the attribute anchors (Ajtop,Ajbot )4j=1 using cosine similarity applies an offset loss that pulls the representation toward directions with low similarity and away from those with high similarity. The r parameter weights each direction to modulate the distortion intensity adaptively toward the direction with higher semantic consensus in each space. The offset loss is defined as follows:

For each attribute, we compute the cosine agreement between the top and bottom anchors in each feature space to obtain an alignment weight wj that reflects attribute-wise divergence. This weight emphasizes attributes that contribute most to the distortion. These weights scale the corresponding offset losses, yielding the final SD attack loss as follows:

3.3 Semantic Direction EOT

Expectation over transformation (EOT) [Yuan et al., 2022] improves robustness to photometric and geometric variations in face-swapping processes and can enhance black-box transferability. However, uniform expectations may dilute informative gradients by averaging them with many weak signals, slowing optimization. In Figure 4, Semantic direction EOT (SD-EOT) replaces the uniform expectation with directionadaptive weighting, precomputing transformation-specific semantic direction vectors in the CI-JES, and assigning stable weights based on their directional conflict with the current attack direction for more robust optimization.

Offline SemanticDirectionEstimation. For each transformation in a fixed bank, a semantic direction vector is estimated to capture the expected shift induced by the transformation in the CI-JES. Following prior EOT settings, a fixed set of differentiable photometric and geometric transformations {Tp}Pp=1 is adopted. For each transformation Tp, the CI-JES shift vector ∆V p x for a face image x is defined by:

where Ejoint(·) extracts an identity embedding (e.g., ArcFace) and a CLIP image embedding from the input image and concatenates them to form the joint representation. The semantic direction vector µp of Tp is defined by:

where D represents a face dataset (e.g., FFHQ), and Ex∼D[·] denotes the expectation over samples from D. The direction vectors µpPp=1 are computed once offline and shipped with the defense, requiring no additional data at deployment time.

Online direction-adaptive optimization. During online optimization, the loss terms for each transformation are adaptively weighted to focus on those that most impede the current attack instance. Given the current adversarial image x∗ , the attack direction in the CI-JES, gadv, is defined as follows:

where LSD-attack denotes the semantic distortion attack loss.

The directional conflict score sp is defined as follows to quantify how a transformation Tp counteracts the current attack direction:

where µp denotes the precomputed semantic direction of Tp, · represents the dot product, and ∥ · ∥2 indicates the ℓ2 norm.

The direction-adaptive weight ap is obtained by applying a softmax normalization over {sp}Pp=1 :

The SD-EOT objective is defined using the weighted expectation of the attack loss over the transformation set:

where L(p) SD-attack denotes the semantic distortion attack loss evaluated on Tp(x∗ ).

3.4 Semantic Robust JPEG

The JPEG compression is common in real-world faceswapping pipelines. Its nondifferentiable quantization suppresses high-frequency perturbations and weakens defense performance. Existing defenses use differentiable JPEG surrogates [Qu et al., 2024] or low-frequency constraints [Jeong et al., 2025]; however, the former may deviate from real JPEG behavior, and the latter can limit semantic disruption. We adopt BPDA [Athalye et al., 2018] with a differentiable surrogate, evaluating the forward objective on real JPEG outputs and approximating gradients through the surrogate.

The real JPEG operator is denoted by Jreal(x) = JPEG(x, q) and its differentiable surrogate is denoted by Jdiff(x) = JPEGdiff(x, q), where the quality factor q is randomly sampled during optimization. In the forward pass, the semantic distortion loss is evaluated on the real JPEG output:

where L(real) SD-attack denotes the semantic distortion attack loss evaluated on Jreal(x∗ ).

In the backward pass, BPDA computes gradients using Jdiff(x∗ ) while maintaining the forward evaluation on Jreal(x∗ ).

3.5 Total Loss Function

This work integrates the three proposed semantic-driven losses into a unified objective function, defined as follows:

Finally, the perturbation θ is updated using the total loss with a learning rate η according to the following equation:

3.6 Generalized Feature Gallery Construction

This work preconstructs a feature gallery to facilitate the perturbation update. The gallery comprises CLIP and identity features for each image. The CLIP features were extracted using Eclip. Prior gallery-based defenses rely on a single identity extractor [Lee et al., 2025], limiting generalization under extractor shifts. Therefore, the identity features were extracted using the SIIR.

Offline Semantic Iterative Identity Refinement. This method extracts an identity feature that remains consistent across identity extractors for proactive defense under extractor shifts. Inspired by cross-model compatibility [Meng et al., 2021], we align embeddings from surrogate extractors used in deepfake pipelines and iteratively consolidate cross-extractor homogeneous components while suppressing extractor-specific variation. As shown in Figure 5, embeddings e1, e2 ∈ R512 are obtained from two frozen surrogates, ArcFace and FaceNet [Deng et al., 2019; Schroff et al., 2015], which cover widely used identity spaces. Two RBT modules [Wang et al., 2020] refine them along separate paths to preserve identity cues and facilitate alignment, producing zn 1 and zn 2 at iteration step n.

Channelwise disagreement dnc and a binary heterogeneous mask mnc are computed at each iteration step:

where H(dn ) denotes the harmonic mean of the channelwise disagreement values {dnc }512 c=1 . This adaptive threshold is recomputed at each iteration to adapt to the changing distribution of channelwise disagreements. Channels selected by mnc are treated as heterogeneous codes and refined in subsequent iterations, whereas channels selected by

˜mnc = 1 - mnc are treated as homogeneous codes and accumulated across iterations to form the homogeneous code accumulation embeddings R1 and R2. The refinement stops when the mask no longer changes between consecutive iterations (mn = mn-1 ), after which R1 and R2 are concatenated and the result is fed into an RBT module followed by a linear projection to obtain the generalized identity feature I ∈ R512 .

This work defines two alignment objectives on the last iteration embeddings zn 1 and zn 2 obtained from the two surrogate paths. First, a statistical alignment loss matches the mean and covariance computed over the minibatch embeddings of zn 1 and zn 2 , where ¯ z1, Σ1 and ¯ z2, Σ2 denote the corresponding mean and covariance, respectively:

Second, contrastive loss pulls the embeddings of the same gallery image closer while separating different gallery images in a minibatch B:

In our setting, we use one gallery image per identity and assign a unique identity index as its label. Here, zn 1,u and zn 2,u denote the last iteration embeddings of the u-th gallery image obtained from the two surrogate paths. Lcontrast encourages cross-path consistency while preserving identity separability.

Finally, the classifier heads of the two surrogate identity extractors are frozen, and I is input, minimizing Lclassification so that both heads predict the identity label for I. Further, Lclassification is defined as the average of two cross-entropy loss functions from the prediction of each head with respect to the identity label. We optimize the gallery objective Lgallery to learn a robust generalized identity feature I, defined as:

4 Experiment

4.1 Experimental Setup

Datasets and Face-swapping Models. This work evaluates the method on two high-quality face datasets, CelebAHQ [Karras et al., 2017] and VGGFace2-HQ [Chen et al., 2023]. The CelebA-HQ dataset contains 30K celebrity images, and VGGFace2-HQ comprises 9,630 identities with about 1.3M facial images, spanning diverse identities, poses, expressions, and illumination. The quantitative evaluation used 1,000 source images, generated 1,000 swaps per set via random source-target pairing, and averaged results across 10 sets to reduce pairing bias. This large-scale random pairing enables statistically grounded evaluation by averaging results over diverse identity combinations and reducing sensitivity to specific pair selections. This work compares the approach against face-swapping models: SimSwap [Chen et al., 2020], FaceDancer [Rosberg et al., 2023], DiffSwap [Zhao et al., 2023], DiffFace [Kim et al., 2025], REFace [Baliah et al., 2025].

EvaluationMetrics. This work assesses visual fidelity and defense effectiveness using five metrics. The peak signalto-noise ratio (PSNR) and structural similarity index measure (SSIM) metrics quantify visual similarity; higher values indicate better fidelity for original-protected pairs, while lower values on the deepfake outputs indicate stronger disruption. † for original-protected comparisons and ‡ for the deepfake images comparisons. The identity score matching (ISM) [Van Le et al., 2023] measures identity similarity between the original and protected images, and defense success rate (DSR) and protection success rate (PSR) [Qu et al., 2024; Lyu et al., 2023] report the rates at which deepfakes fail to preserve the source identity and protected sources are not recognized as the original identity, respectively. In all tables, the best results are in bold, and the second-best are underlined.

Baselines. This work compares SGD-Guardwith six proactive defenses: Anti-Forgery [Wang et al., 2022], CMUAWatermark [Huang et al., 2022], DF-RAP [Qu et al., 2024], DiffAM [Sun et al., 2024], FaceShield [Jeong et al., 2025], and SCOL [Lee et al., 2025]. The DiffAM is makeup-based. Anti-forgery, CMUA-Watermark, DF-RAP, and FaceShield perturb protected images, whereas SCOL performs fusion in the latent space and injects adversarial watermark-like perturbation. For all noise-based methods, ϵ is set to 0.05 [Jeong et al., 2025] under the ℓ∞ norm for fair comparison. For optimization or generation based baselines, we follow the hyperparameters and protocols specified in the respective papers.

Implementation Details. The feature gallery stores paired generalized identity features and CLIP features. This work adopts FaRL as the CLIP image encoder Eclip for face analysis [Zheng et al., 2022]. The gallery is precomputed offline and occupies only 72 MB, as it stores compressed feature embeddings rather than raw image data, enabling easy reproducibility and deployment. As described in Section 3.2, we set the retrieval parameter to k = 10 throughout. For semantic-direction EOT, the direction vectors are estimated using the full FFHQ dataset. Details on k are provided in Appendix B, and the algorithms for offline feature gallery building and online SGD-Guard are summarized in Appendix D.

4.2 Main Results

Evaluation of Deepfake Results. Table 1 compares proactive defenses on five face-swapping models across two datasets. Anti-Forgery and CMUA achieve low DSR, while DF-RAP, DiffAM, and FaceShield show high variance. In contrast, SGD-Guard is the most consistent, reaching top DSR in most cases and the lowest PSNR and SSIM on deepfake outputs. Figure 6 qualitatively compares SGD-Guard with DiffAM, FaceShield, and SCOL, which are used as state-of-the-art baselines in later experiments, and also evaluates protection against ReFace. DiffAM often causes large facial changes, FaceShield adds visible noise, and SCOL can blur protected images. In contrast, SGD-Guard better disrupts identity cues and preserves natural visual quality. Additional visual results are included in Appendix C.

Evaluation of Protected Images. Table 2 compares the protected-image quality and identity suppression. Antiforgery preserves PSNR and SSIM but yields a high ISM value, indicating weak identity protection, whereas DiffAM lowers the ISM value at the cost of severe visual degradation. In contrast, SGD-Guard achieves the best PSNR and SSIM with the lowest ISM value, offering substantial identity disruption without sacrificing visual fidelity.

Generalizability. Figure 7 compares DiffSwap, DiffFace, and ReFace under face-swap model shifts. The first row shows swaps from source images, and the second shows swaps from protected sources. Across all models, SGDGuard consistently degrades synthesized outputs, indicating robust protection under model changes. Table 3 evaluates identity extractor shifts using four face verifiers. Methods optimized for ArcFace achieve high PSR on it but drop by over 45%p on average on other verifiers. In contrast, SGDGuard leverages ArcFace and FaceNet surrogates, achieving the highest PSR while limiting the average gap to 26%p on VGGFace and SphereFace. These results suggest transfer beyond the surrogate pair, although they do not guarantee coverage of proprietary identity extractors whose embedding distributions substantially differ from the evaluated verifiers.

Robustness to Post-processing Attacks. Table 4 evaluates robustness to standard post-processing on CelebA-HQ using REFace. We apply diffusion purification [Nie et al., 2022], JPEG compression with random quality (q ∼ U[1, 70]), and crop/rotate transforms, since vulnerability varies by transformation. Following [Yuan et al., 2022], images are randomly cropped to retain at least 90% area and rotated up to 15◦ . Across all settings, SGD-Guard is the most robust, achieving the highest DSR and lowest PSNR, indicating disruption that persists after post-processing. Additional evaluations under challenging conditions (e.g. real-world extreme poses and various resolutions), with qualitative robustness results, are provided in Appendix C.

Ablation Study. Table 5 presents an ablation study on the distortion and generalization performance of SD-attack and SIIR using REFace on the VGGFace2-HQ dataset. SD-attack achieves a low SSIM regardless of its integration with the SIIR module, signifying substantial distortion in the protected images. When utilized alongside the SIIR model, it demonstrates high DSR across various backbones and exhibits robust generalization performance.

Table 6 ablates the robustness of SD-EOT and SR-JPEG to post-processing under the same setting as Table 5. Using only SD-EOT achieves robustness to crop-and-rotate transformations, while adding SR-JPEG strengthens robustness to JPEG compression. Combining SD-EOT and SR-JPEG yields the most reliable protection, maintaining consistently low SSIM and high DSR values under various post-processing variants.

Complexity. Figure 8 shows SGD-Guard is the most efficient defense in FLOPs, parameters, and inference time. DiffAM is costly due to diffusion inversion and repeated UNet passes, while FaceShield increases cost by optimizing diffusion cross-attention. In contrast, SGD-Guard freezes CLIP/identity encoders and optimizes only a compact pixelspace perturbation with similarity-based objectives. It uses a single-step latent diffusion purifier with LoRA adapters [Lei et al., 2025] and reduces inference time by parallelizing computation and precomputing SIIR and SD-EOT directions offline. A discussion of limitations is provided in Appendix E.

5 Conclusion

This study proposes SGD-Guard, a proactive defense strategy that provides robust, generalized protection against faceswapping deepfakes. By integrating facial attribute and identity spaces, SGD-Guard identifies distortion directions and learns perturbations that steer representations toward a protected state. Furthermore, diffusion-based purification and post-processing techniques prevent feature shifts in the latent space to counteract attacks intended to neutralize perturbations, ensuring resilience against image transformations. The proposed method achieves generalized defense across diverse model backbones in a time-efficient manner.

Acknowledgements

This work was supported by the IITP grant funded by the Korea government (MSIT) (RS-2022-00156287, RS-202300256629, RS-2024-00437718, RS-2026-25619158).

Contribution Statement

Seung-hyeok Back and Do Hyun Ki contributed equally to this work.