Parameter-Efficient Dual-Loss Adaptation with Logit Divergence: A Unified Approach for Adversarial Example Detection and Robust Inference

1 Introduction

Deep neural networks (DNNs) underpin modern computer vision systems in safety-critical applications such as autonomous driving, medical imaging, and content moderation. Despite their strong performance, DNNs remain vulnerable to adversarial examples, where imperceptible input perturbations can induce incorrect predictions. In practical deployments, the challenge is not only to improve average-case accuracy, but also to make inference trustworthy under worstcase perturbations while respecting real-world constraints on latency, power, and computational complexity.

Existing defenses largely follow two approaches. Adversarial example detection (AED) aims to identify adversarial inputs at inference time and then reject, defer, or route them to a conservative pipeline. AED systems often introduce extra inference steps or additional detector modules, require dataset- or deployment-specific calibration, and can be fragile under adaptive attacks where the detector becomes part of the attack objective [Xu et al., 2018; Metzen et al., 2017; Liang et al., 2018; Carlini and Wagner, 2017]. Robustness enhancement methodologies aims to preserve correct predictions within a perturbation budget by switching training objectives, regularizing or stabilizing inputs and activations, or enhancing model structural complexity. They are often costly to train, may compromise clean accuracy, and are not readily applicable to third-party models without retraining or structural changes [Madry et al., 2018; Shafahi et al., 2019; Tsipras et al., 2019; Zhang et al., 2019]. As a result, detection and robust inference are often implemented as separate workflows, further increasing engineering complexity and inference overhead when both are required [Meng and Chen, 2017].

To bridge detection and robustness, Reverse Cross-Entropy (RCE) [Pang et al., 2018] has been explored as a principled mechanism to reshape logit geometry and improve separability for anomaly detection. The RCE-KDE paradigm, which trains a classifier with RCE and then attaches a kernel density estimation (KDE) detector, can yield strong detection performance under controlled settings. Nevertheless, it exhibits two practical obstacles. First, RCE optimization can be unstable and becomes harder to scale as the number of classes increases. Second, it requires full retraining of the target model and relies on an external KDE module at inference time, which incurs storage cost, calibration effort, and additional deployment complexity. Recent work mitigates the retraining requirement by integrating stabilized RCE variants such as RCE with temperature scaling and weight focusing (RCE-TF) [Fu and Donato, 2025] coorindated with parameter-efficient adapters. However, this line of work still falls short of a detector-free and deployment-friendly unification of detection and robust inference.

This work departs from model architectures using external detectors and instead treats disagreement among adapters trained for different purposes as an endogenous adversarial detection signal. Loss functions and training objectives induce different decision boundaries and logit geometries. As a result, adapters trained under different losses and objectives tend to agree on the predicted label for benign inputs. However, the same set of adapters tends to disagree in systematic ways under adversarial perturbations. Therefore, we propose an inter-adapter logit divergence as an agreement-based threat score to select the most reliable prediction among adapters at inference time. In this way, detection and defense can be unified into a single inference primitive without an extra detector module.

In summary, we propose D3Adapter, a Divergence-aware Detection and Defense adapter framework that unifies AED and robust inference. D3Adapter attaches a small library of lightweight adapters to a frozen backbone; adapters are trained under two loss families (CE and RCE-TF, an optimized version of RCE) to target complementary objectives (clean training and adversarial training) with the goal of producing different logit behaviors. During inference, D3Adapter performs a single-pass forward computation through all adapters, quantifies inter-adapter divergence to obtain a threat score, and outputs the prediction from the appropriate adapter based on the outcome of the adversarial detection. The resulting overhead is predictable and scales linearly with the number of adapters, making the framework suitable for resource-constrained deployment. Our main contributions are as follows:

• Unified inference primitive for detection and defense: We introduce an agreement-based threat score via interadapter logit divergence and a same-pass routing rule that jointly provides adversarial example detection and robust inference without an external detector module.
• Threat-aware robustness under strong attacks: We evaluate D3Adapter under transfer-based and adaptive white-box attacks, demonstrating improved detection quality and robust accuracy while preserving clean performance.
• Scalability and deployment efficiency: We study scalability across datasets with varying numbers of classes, and show that unified detection and robust inference can be achieved with overhead proportional to the number of adapters. The framework also supports controlled ablations through interchangeable loss families, divergence metrics, and selection rules.

2 Background and RelatedWork

This section reviews four threads that motivate D3Adapter: (I) loss-based logit geometry shaping, (II) parameter-efficient fine-tuning, (III) threat models and robust evaluation practices, and (IV) divergence-based adversarial detection.

2.1 Loss Shaping for Robustness and Detectability

Standard training typically minimizes cross-entropy, encouraging highly confident predictions. Such overconfident logits can worsen calibration and amplify sensitivity to label noise and distributional outliers, motivating alternative losses that reshape probability geometry. Common strategies include probability tempering and smoothing, e.g., temperature scaling for calibration [Guo et al., 2017] and label smoothing [Szegedy et al., 2016], as well as reweighting hard examples such as focal loss [Lin et al., 2017].

TRADES [Zhang et al., 2019] is a representative surrogateloss design that trains the model on adversarial examples but optimizes it with a tailored clean-and-adversarial target for clean-robust performance trade-off. Reverse Cross-Entropy (RCE) [Pang et al., 2018] is proposed as a robust training objective to suppress incorrect representations, opposite to normal cross-entropy (CE) loss. RCE enables thresholding- and density-based out-of-order distribution detection. It is also used as a noise-tolerant counterpart of CE loss in Symmetric Cross Entropy [Wang et al., 2019] to stabilize noisy label training. Recent work further modifies RCE-style objectives to improve adversarial robustness and scale on larger label spaces [Fu and Donato, 2025]. It couples optimized RCETF with lightweight adapters (introduced in Section 2.2) while the backbone model is frozen, avoiding full-model retraining.

These losses can be incorporated with adversarial training, a common method for robustness enhancement, as an alternative objective in the training process to further increase model's adversarial robustness thanks to their distinguished logits distribution. These collective efforts can deliberately shape model's logit geometry and representation structure, yielding complementary behaviors in calibration, density separation, and adversarial smoothness. We exploit these controllable behaviors to construct multiple, measurable prediction profiles for threat scoring and robust inference, and instantiate them via parameter-efficient fine-tuning, introduced in Section 2.2.

2.2 Parameter-Efficient Fine-Tuning with Adapters

Full-model fine-tuning is often memory- and computeintensive. Parameter-Efficient Fine-Tuning (PEFT) addresses the problem by keeping a backbone frozen and training small additive modules. In computer vision domain, residual adapters [Rebuffi et al., 2017] were introduced to steer a shared ResNet [He et al., 2016] backbone towards correct inference over multiple domains via lightweight residual branches. Houlsby et al. introduced bottleneck adapters in a transformer architecture, allowing new language tasks to be added without retraining the full model [Houlsby et al., 2019]. This modularity naturally enables maintaining an adapter library and dynamically selecting or composing modules: AdapterHub [Pfeiffer et al., 2020] demonstrates a practical ecosystem of pre-trained adapters, while AdapterFusion [Pfeiffer et al., 2021] shows that multiple pretrained adapters can be combined via a lightweight fusion mechanism without modifying the backbone. Other PEFT families such as LoRA inject low-rank updates into linear layers [Hu et al., 2022]. Although LoRA-style low-rank adaptation can be extended beyond Transformer models, we focus on adapter modules because they better support a compact library of independently trained adapter candidates for our victim ResNet-based model.

The paradigm of swapping adapters on a frozen backbone is particularly attractive for deployment as it enables multitask adaptation without the need of replacing entire models. MEMTI [Donato et al., 2019] and Adapter-ALBERT [Fu et al., 2024] have shown that adapter-equipped networks can improve on-device efficiency by selectively updating only a small parameter subset in faster memory tiers. This modularity and previous AdapterHub result a key enabler for our work: maintaining a small adapter library with deliberately different robustness profiles makes it feasible to both measure disagreement and route to a robustness-oriented adapter under attack.

2.3 Adversarial Attacks and Evaluation

Adversarial attacks are commonly categorized by perturbation norm. L∞ attacks include one-step FGSM [Goodfellow et al., 2015] and multi-step PGD [Madry et al., 2018], the latter serving as a strong baseline for first-order robustness. In this work, we will primarily use PGD as the main adversarial attacking algorithm. L2 attacks are typified by C&W [Carlini and Wagner, 2017], and L1 attacks include the elastic-net formulation EAD [Chen et al., 2018]. Adversarial attacks can also be categorized by attacker's knowledge of the victim model. Transfer-based (black-box) attacks are naive perturbations that are generated by the attacker from the victim model and applied to other similar models. Adaptive (whitebox) attacks are conducted when the attacker has knowledge to the victim model and can access its gradients and defense modules, if applicable.

A key lesson from prior work is that detection-style defenses are vulnerable under adaptive attackers: many detectors can be bypassed by optimizing an attack objective tailored to the detector [Carlini and Wagner, 2017], and defenses relying on gradient masking can appear robust while failing under stronger attacks [Athalye et al., 2018]. We therefore report results under both transfer and adaptive settings for a comprehensive performance evaluation of our framework. We also craft a detector-aware adaptive PGD attack to optimize both the model itself and its detection mechanism to stress our divergence threat-scoring signals.

2.4 Divergence-Based Adversarial Detection

Multiple lines of work indicate that disagreement between predictive distributions correlates with epistemic risk under distribution shift. Deep Ensembles exploit member disagreement for uncertainty estimation [Lakshminarayanan et al., 2017], and Bayesian approximations such as MC Dropout use stochastic forward passes to estimate predictive uncertainty [Gal and Ghahramani, 2016]. Consistency regularization methods such as AugMix [Hendrycks et al., 2020] use Jensen-Shannon divergence (JSD) [Lin, 1991] across augmented views to encourage stable predictions.

Taken together, prior uncertainty and consistency literature supports using distributional disagreement as a risk proxy, and divergences such as JSD and symmetric KL (sKL) provide simple, fully differentiable summary statistics on predictive distributions. Motivated by this principle, we use an inter-prediction divergence score as the threat signal in our framework; the exact formulation and ablations are presented in Section 4.

3 Threat-Aware Adapter Library for One-Pass Detection and Defense

To unify AED and robust inference within a single inference procedure, D3Adapter relies on a small library of lightweight adapters attached to a frozen ResNet-18 backbone. The library is intentionally diversified along two axes: loss geometry (CE and RCE-TF) and training objective (clean training and adversarial training). This section first explains the construction of adapter library from an observation of RCE-TF clean performance recovery and the library's behavior under transfer-based and adaptive PGD attacks. The resulting robustness and failure mode divergence later becomes an intrinsic signal for one-pass detection and routing.

3.1 Balancing CE and RCE-TF Clean Accuracy

We start from a ResNet-18 backbone pretrained on ImageNet and extended with bottleneck adapters. For each downstream dataset, we freeze the backbone and train only the adapters

(a) Transfer PGD (CE to RCE) Attack

(b) Adaptive PGD Attack

51.2

Clean Acc.

CA-CE

CA-RCE

26.2

33.3

19.8

23.3

10.5

10.1

3.5

CIFAR-10 SVHN GTSRB CIFAR-100

Dataset

Figure 2: Robust accuracy comparison of CA-CE (blue) and CARCE (red) adapters under (a) transfer PGD attack crafted from CE model and (b) adaptive PGD attack crafted by their own loss functions. The dashed green line above each data is the clean accuracy of the corresponding model.

and the dataset-specific classification head. This setting keeps the adapter library lightweight, while ensuring that different candidates share the same feature extractor, which is required by a unified inference path.

A practical requirement is that the clean accuracy remains usable after introducing RCE-TF, since the library must serve both benign and adversarial inputs. Prior RCE-TF work [Fu and Donato, 2025] reports clean CIFAR-10 and CIFAR-100 accuracies with adapter size 64. In our adapter-only training regime, we start with adapter size 128 to improve accuracy and adjust to 256 when RCE-TF accuracy lags behind the CE counterpart. We explicitly measure the clean accuracy versus parameter overhead trade-off across datasets with different numbers of classes, shown in Table 1. It compares CE and RCE-TF adapters on CIFAR-10, SVHN, GTSRB, CIFAR100, and Tiny-ImageNet, covering class counts from 10 to 200. The RCE-TF adapter outperforms CE adapters with the same size setting on datasets with up to 100 classes. Additionally, the parameter overhead grows proportional to the adapter size and number of classes of the dataset. The overhead becomes non-negligible when adapter size increases to 256, especially on Tiny-ImageNet where the result is atypical in that an RCE-TF adapter with size 256 still underperforms a CE adapter with size 128. On Tiny-ImageNet's CE adapters, increasing adapter size to 256 also does not produce a significant accuracy improvement. This outcome suggests that increasing adapter capacity alone is not sufficient to close the gap in the 200-class regime under a frozen backbone. A plausible explanation is that RCE-TF relies on a reversed logit geometry and a strong separation between the true class and many non-target classes. When the number of classes is larger, the CE/RCE-TF competition among their own targeted classes becomes more significant, and adapteronly training may not have enough strength to reshape representations to the extent required by the RCE-TF objective. In addition, Tiny-ImageNet differs from the ImageNet pretraining distribution and resolution, which can make adapter-only adaptation more sensitive to the choice of loss geometry. In our setting, this combination can limit the benefit of RCE-TF even when the adapter size is increased.

To stay within a conservative overhead and accuracy scenario, we use adapter size 128 as the default and instantiate two clean-trained adapter candidates: a clean-trained adapter by CE loss (CA-CE) and a clean-trained adapter by RCE-TF loss (CA-RCE). On CIFAR-100, CA-CE achieves 72.76% clean accuracy, while CA-RCE achieves 77.86%. For later experiments we focus on CIFAR-10, SVHN, GTSRB, and CIFAR-100, and we exclude Tiny-ImageNet due to its different scaling behavior under adapter-only training.

3.2 Transfer and Adaptive PGD Threats: Attack Protocols and Threat Signals

We evaluate the adapters under two threat models that cover both black-box and white-box adversarial attacks.

Transfer-based black-box PGD. In the transfer setting, the attacker crafts adversarial examples on a surrogate model and applies them to the target model without access to the target parameters. Figure 2(a) shows the accuracy of CA-CE and CA-RCE adapters when subjected to a transfer PGD attack with strength ε = 8/255 crafted from the CE model. CA-RCE adapter has shown stronger resilience on the four datasets than the CA-CE counterpart.

Adaptive white-box PGD. In the adaptive setting, the attacker has full access to the target model and adapts the attack to the target loss. For each target, we craft adversarial examples by directly optimizing the corresponding loss used by the target so that the attack is loss-consistent with the target geometry. This setting provides a stricter evaluation than transfer attacks because it removes the surrogate mismatch. Figure 2(b) shows the accuracy of CA-CE and CARCE adapters under adaptive PGD attack of the same strength as Figure 2(a). While CA-RCE still shows stronger robustness than CA-CE adapter, both clean-trained candidates are not resilient enough and lose practical reliability under lossmatched adaptive attacks. This necessitates adversarial training to enhance adapter candidates.

Robust candidates and divergence trends lead to Divergence AED. To obtain candidates that remain functional under adaptive attacks, we perform PGD adversarial training and obtain two additional adapters: a CE robust adapter (RA-CE) and an RCE-TF robust adapter (RA-RCE), under adaptive PGD ε = 8/255. Table 2 shows all four candidates' clean and robust accuracies under adaptive PGD ε = 8/255 in CIFAR-10, SVHN, GTSRB, and CIFAR-100 datasets.

Additionally, Figure 3 shows the robust accuracies of four pairs of RA-CE and RA-RCE adapters, each pair trained under PGD ε = {2, 4, 8, 16}/255, under transfer PGD attack with strength from 0 to 32/255. Noticeably, CE and RCE-TF adapters exhibit a systematic performance separation under the same perturbations: on benign inputs the heads tend to agree (left side of the X-axis), while the separation and disagreement become more pronounced as perturbation strength increases.

In summary, the four adapters from different loss-objective correlations exhibit different characteristics and strengths under clean and adversarial settings, and generates unique logits. No single candidate simultaneously optimizes clean utility, transfer robustness, and loss-matched adaptive robustness. This trend suggests that cross-adapter differences in logits and predictions can serve as an intrinsic signal for AED, and Section 4 shows how this signal can guide defense decisions by measuring inter-adapter divergence.

4 Divergence-Aware Adapter Detection Mechanism

A representative RCE-style detection baseline is RCEKDE [Pang et al., 2018], which pairs an RCE (or RCEvariant) classifier with a KDE detector on pre-classification features. This design introduces an extra inference-time scoring module, which requires constant detector-specific calibration on KDE's bandwidth and threshold throughout the inference period, and expands the differentiable attack surface when the detector is included as objective for adaptive attacks. D3Adapter avoids an external detector and instead uses model-intrinsic disagreement signals produced by the adapter library, enabling one-pass scoring and robust prediction.

4.1 Inter-Adapter Calibration and Divergence Score

All four adapters run on a single frozen ResNet-18 backbone and output logits zi(x) ∈ RC for any given input x, where the subscript i refers to the i-th adapter and C is the number of classes in the current dataset. Adapter outputs must be normalized before any divergence measurement because CE and RCE-TF induce opposite logit orientations, and because different training objectives can yield different confidence scales.

Logit orientation. To enforce a consistent semantic direction where larger values indicate higher confidence, we apply a loss-family dependent orientation transform:

Temperature scaling. Even after orientation, adapter confidences are not directly comparable. We therefore fit a peradapter temperature τi on a small clean calibration split by minimizing cross-entropy: Following the concept of temperature scaling [Guo et al., 2017], the calibrated posterior is pi(x) = softmax(˜ zi(x)/τi).

This temperature scaling calibration aligns predictions from different adapters so that divergence values reflect disagreement rather than scale artifacts. The calibration happens before the inference process and the same values are used for all other normalization process. This differs from KDErelated calibration, where frequent sampling for bandwidth selection and density-threshold selection are required to make feature-space density estimates operational as an anomaly score.

Divergence computation. Given calibrated posteriors, we compute pairwise divergences across adapters. For each (i, j) pair of adapters, we compute JSD:

where m = 1 2 (pi + pj) and additionally a sKL term:

With four adapters, this yields 6 pairwise JSD values and optionally 6 sKL values. To capture the structured separation observed in Section 3, we also compute group-level statistics, including mean divergence between the CE-family and the RCE-family, and within-family divergence between cleantrained and robust-trained variants.

Score fusion. All divergence features are standardized using the same data for temperature scaling calibration. The normalized features are then aggregated into a scalar divergence score s(x). We use equal-weight fusion as a default:

and optionally consider a learned logistic fusion:

where (w, b) are trained on a balanced clean/adversarial subset. Figure 4 shows a performance comparison between fusion modes on clean data, transfer PGD, adaptive PGD, and detector-aware (DA) adaptive PGD attack which will be addressed in Section 4.3 as a stress test to our detection system.

4.2 Robust Prediction Routing Based on Threshold

Unifying detection and robust inference without introducing an extra inference module is beneficial for deployment. D3Adapter uses divergence score s(x) to both detect suspicious inputs and select which adapter logits to trust for the final prediction.

There are two thresholds for the system: θsus (suspicious) and θrej (rejection). The thresholds are defined using the clean-score quantiles Q from the same calibration data used for temperature scaling and divergence feature standardization:

with typical q = 0.98 for θsus and q = 0.995 for θrej. This avoids tuning thresholds on a specific attack configuration.

Given s(x) and thresholds (θsus, θrej), we perform onepass routing:

• If s(x) ≤ θsus: treat x as benign and output the prediction from CA-RCE, which has the highest clean accuracy among the four adapters according to Table 2.
• If θsus < s(x) ≤ θrej: treat x as suspicious and output the robust-trained candidate (RA-RCE) with higher confidence.
• If s(x) > θrej: optionally abstainwhen even the selected robust adapter has low confidence. The rejection rate varies according to selected fusion methods and attacking schemes, depicted in Figure 4.

This design directly reuses the adapter logits computed for divergence scoring, and does not require any auxiliary detector forward pass.

4.3 Detector-Aware Adaptive Attack Evaluation

Detection-style defenses must be evaluated under adaptive attacks that differentiate through the full pipeline [Tram`er et al., 2020]. We therefore adopt a detector-aware adaptive PGD objective that jointly maximizes misclassifications while minimizing the divergence score:

where ˜ zi and τi follow Eq. (1) and temperature scaling, and γ trades off detector evasion against misclassification. We use multi-restart PGD with 5 restarts, 40 steps each, and retain the perturbation with the highest joint objective per sample.

4.4 Controlled Comparison with KDE Detection

For a controlled baseline, we implement a Gaussian KDE detector on the pre-classification features of the frozen backbone with a single adapter. Given clean training features {fk}nk=1 ⊂ Rd , the KDE density estimate at test feature f is

with bandwidth h chosen by Silverman's rule [Silverman, 1986]. The detection threshold is set at the 5th percentile of training log-densities.

From Figure 5, under detector-aware PGD, where the attacker jointly minimizes the divergence score and maximizes misclassifications, D3Adapter retains strong separation (AUROC > 0.97) on CIFAR-10, SVHN, and GTSRB, whereas KDE degrades to near-random on SVHN (0.52) and remains below 0.65 on CIFAR-10. This gap reflects the structural difficulty of simultaneously suppressing disagreement across four intentionally diverse adapters while also inducing misclassifications.

4.5 Reduced-Adapter Divergence Design

Since executing multiple adapters incurs both parameter and latency overhead, a natural question is whether a smaller adapter set can preserve detection strength. We construct a three-adapter variant by removing RA-CE, the adapter with the narrowest applicable scenario based on the performance observed in Section 3. All other pipeline components, including divergence definition, score aggregation, and threshold calibration, remain unchanged.

Figure 6 compares the three- and four-adapter configurations for D3Adapter, and KDE. On CIFAR-10, the threeadapter variant achieves AUROC of 0.89 under transfer PGD, 0.95 under adaptive PGD, and 0.92 under detector-aware PGD, all within 0.06 of the four-adapter baseline. On GTSRB, the three-adapter variant reaches 0.99 AUROC under adaptive PGD, slightly exceeding the four-adapter result. However, under detector-aware PGD, the four-adapter configuration consistently outperforms the three-adapter variant: AUROC differences are 0.05 on CIFAR-10, 0.11 on SVHN, and 0.06 on GTSRB. This suggests that the fourth adapter,

D3Adapter vs KDE Detection Performance

Three-Adapter vs KDE Detection Performance

despite limited standalone utility, provides an additional constraint that the attacker must satisfy when optimizing against the full detection objective.

5 Conclusion

D3Adapter is a threat-aware framework that unifies AED and robust inference within single forward computation on a frozen backbone, using a small library of lightweight adapters trained under CE and RCE-TF losses with clean and adversarial objectives. Inter-adapter logit divergence provides an agreement-based threat score without any external detector and also guides adapter selection for robust inference. D3Adapter achieves competitive detection quality and robust inference performance under transfer-based and adaptive white-box attacks.

Future work will focus on reducing runtime cost via adapter subset selection and layer-local deployment, improving the calibration of inter-adapter logit divergence for broader class scales, and extending the framework to additional threat models.