PILO: Principal Component-based Implicit Regularization with Low-rank Optimization for Robust Transfer Learning

1 Introduction

The transfer learning paradigm [Yosinski et al., 2014; Howard and Ruder, 2018; Devlin et al., 2019; Raffel et al., 2020]-comprising pre-training on large-scale datasets and subsequent task-specific fine-tuning-has become a cornerstone of modern machine learning.

∗Corresponding author. taixiangjiang@gmail.com

With the rise of foundation models [Bommasani et al., 2021], fine-tuning pre-trained feature extractors enables stateof-the-art performance across a multitude of downstream tasks. Empirical evidence from computer vision confirms that initializing weights from pre-trained models consistently surpasses random initialization [Kornblith et al., 2019; He et al., 2019]. However, this success is critically undermined by the models' vulnerability to adversarial perturbations, a flaw that poses significant risks in high-stakes applications like autonomous driving [Eykholt et al., 2018; Kurakin et al., 2018] and medical image analysis [Buch et al., 2018]. This necessitates the development of robust finetuning methodologies to ensure model resilience in securitysensitive settings.

To this end, various robust fine-tuning techniques have emerged. Adversarial Training (AT), which incorporates PGD-generated examples [Madry et al., 2017], is a foundational method but often impairs accuracy on clean data. Subsequent approaches like TRADES [Zhang et al., 2019] achieved a better balance, while TWINS [Liu et al., 2023] further enhanced performance using dual batch normalization architectures [Xie et al., 2020]. Most recently, AUTOLORA [Xu et al., 2023] introduced a dual-branch structure to mitigate gradient conflicts. Despite these advances, these methods are all built on a full-parameter fine-tuning strategy. When applied to downstream tasks with limited data, this approach often leads to a critical failure mode we term catastrophic forgetting of robustness, where the generalizable robust features learned during pre-training are overwritten, resulting in severe robust overfitting [Kumar et al., 2022; Rice et al., 2020]

In this work, we argue that this failure stems from a flawed monolithic assumption. We posit a new principle: adversarial vulnerability is not diffuse but is disproportionately concentrated in the principal components of pre-trained weight matrices. To validate this, we conducted a granular spectral ablation study with identical rank budgets. As shown in Table 1, fine-tuning strictly the top-r components (Topr) consistently outperforms the middle and bottom counterparts and even surpasses full-parameter fine-tuning. This suggests that lower spectral components likely harbor taskirrelevant noise that exacerbates robust overfitting. These high-magnitude spectral components represent the most sensitive directions of the feature space, which we hypothesize are the primary targets of adversarial attacks. This principle offers a direct solution to the gradient instability plaguing prior methods; as shown in Figure 1, isolating optimization to the principal components yields a remarkably high and stable gradient similarity between the competing objectives, in stark contrast to the conflicts observed in full-parameter approaches.

Building on this insight, we introduce Principal component-based Implicit regularization with Low-rank Optimization (PILO), a novel framework that operationalizes our principle through a structurally asymmetric dual-branch architecture (as shown in Figure 2). PILO decouples the learning objectives:

1. An adversarial branch performs a targeted update exclusively on the principal components, initialized via SVD [Meng et al., 2024]. This acts as a powerful form of implicit regularization, anchoring the model to its robust pre-trained state and preserving its core robustness, an idea motivated by prior work on robust transfer [Shafahi et al., 2019].
2. A complementary natural branch employs a standard low-rank adapter to gently fine-tune for clean accuracy, which is discarded during inference to maximize efficiency.

As illustrated in Figure 3, fine-tuning only the principal components of a ResNet18 model with TRADES already surpasses a fully fine-tuned baseline across five datasets, confirming the fundamental benefit of our targeted strategy. Our key contributions are as follows:

• We address the problem of robust overfitting by introducing a parameter-efficient fine-tuning strategy focused solely on the principal components of pre-trained weights. This approach imposes strong implicit spectral regularization, improving generalization on limited downstream datasets.
• We advance the resolution of gradient conflicts through a novel dual-branch architecture that strategically separates optimization paths: a spectrally-guided branch for adversarial objectives and a complementary low-rank

module for natural objectives.

• Extensive experiments demonstrate that PILO significantly improves model robustness on downstream datasets compared to state-of-the-art methods. In contrast to prior full-parameter approaches, our method achieves this with only about 29-40 % of the trainable parameters.

2 RelatedWork

2.1 Parameter-Efficient Fine-tuning

Parameter-efficient fine-tuning (PEFT) adapts large pretrained models to downstream tasks by updating only a small subset of parameters, significantly reducing storage and computational overhead. Representative strategies include adapter-based methods [Houlsby et al., 2019; Pfeiffer et al., 2020; Rebuffi et al., 2017], which insert lightweight trainable modules into transformer layers, and prompt-based techniques [Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021] that prepend learnable vectors to input sequences. Among these, Low-Rank Adaptation (LoRA) [Hu et al., 2022] has gained prominence by freezing the pretrained backbone and injecting rank-decomposition matrices, ensuring zero additional inference latency. However, LoRA's standard random initialization ignores the structural information of original weights, potentially hindering optimization. PISSA [Meng et al., 2024] addresses this by initializing adapters via principal component decomposition. This spectrally-guided approach better preserves the pretrained capabilities at initialization, leading to faster convergence and superior performance.

2.2 Adversarial Attacks

The field of adversarial attacks originated with the Fast Gradient Sign Method (FGSM) [Goodfellow et al., 2014]. Subsequently, Projected Gradient Descent (PGD) [Madry et al.,

2017] became the standard for multi-step optimization, while C&W [Carlini and Wagner, 2017] introduced high-quality optimization-based attacks. To ensure reliable evaluation, AutoAttack (AA) [Croce and Hein, 2020b] was developed as a parameter-free ensemble incorporating diverse strategies like FAB [Croce and Hein, 2020a] and Square Attack [Andriushchenko et al., 2020]. In black-box scenarios where gradients are unavailable, methods rely on zeroth-order optimization like ZOO [Chen et al., 2017] or leverage the transferability of adversarial examples [Papernot et al., 2016].

2.3 Robust Fine-Tuning for DownstreamTasks

Following foundational work on the transferability of robust features [Salman et al., 2020; Shafahi et al., 2019; Hendrycks et al., 2019], research into robust fine-tuning has grown substantially. While it was shown that robustly pretrained models can improve clean accuracy on downstream tasks [Salman et al., 2020], directly transferring robustness requires specialized fine-tuning. Conventional adversarial training is the most straightforward approach, but it often incurs high computational costs and degrades performance on natural examples. This trade-off catalyzed more sophisticated methodologies.

Subsequent research has formulated robust fine-tuning as a multi-objective optimization problem. TRADES [Zhang et al., 2019] pioneered a regularization-based approach to explicitly balance standard and adversarial performance. TWINS [Liu et al., 2023] improved upon this by addressing the distribution shift between natural and adversarial examples with a dual batch normalization (BN) architecture. More recently, AUTOLORA [Xu et al., 2023] introduced a dualbranch architecture to decouple the competing objectives and stabilize optimization. Despite these architectural advances, these methods still rely on full-parameter updates or generic PEFT approaches, which can lead to the catastrophic forgetting of robustness on limited data.

In contrast to these methods, our work is founded on a more fundamental principle. Instead of only decoupling the optimization objectives architecturally, PILO performs a structural decoupling at the parameter level. By executing a surgical, spectrally-guided update on the principal components, i.e., the hypothesized locus of vulnerability, while using a distinct low-rank adapter for natural data, PILO introduces a principled, asymmetric fine-tuning strategy that previous methods lack.

3 Methodology

Building on the challenges identified in the introduction, we propose a parameter-efficient fine-tuning strategy that selectively updates principal components while employing a dualbranch optimization approach. The following sections detail each component of our strategy.

3.1 The Spectrally-Guided Adversarial Branch

For the adversarial objective, we design a branch that directly targets the hypothesized locus of vulnerability: the principal components of the pre-trained weights. This design is motivated by the principle that adversarial perturbations primarily exploit the most sensitive directions in the weight space, which correspond to the largest singular values. By initializing this branch using singular value decomposition (SVD), we perform a surgical update on these critical directions, providing a more effective and direct path to enhancing adversarial robustness.

The trainable matrices Badv ∈ Rm×radv and Aadv ∈ Rradv×n are initialized directly from the SVD of the pre-trained weight matrix Wpre = USV⊤:

Here, radv is the rank of the branch, determining the number of principal components to fine-tune. The forward propagation for an adversarial example x∗ is then formulated as:

where Wres = U[:, radv :]S[radv :, radv :]V⊤[:, radv :] is the residual matrix containing the less significant, frozen components of the original weight. This decomposition constrains all parameter updates to the principal component subspace, acting as a form of implicit spectral regularization. This anchors the fine-tuned model to the robust representations of the pre-trained model, preventing catastrophic forgetting of robustness.

The adversarial input x∗ is generated using a standard attack method:

where F (·) is the model, δ is the adversarial perturbation constrained by the Lp-norm, and ϵ denotes the adversarial budget.

3.2 The Auxiliary Natural Branch for Adaptation

To maintain accuracy on clean data, we employ a second, lightweight branch for the natural objective. This branch uses a standard low-rank adaptation approach, initialized for gentle, gradual learning. The matrices are defined as:

where r is the rank of the natural branch. Initializing matrix B with small random values and matrix A to zero ensures that the model's performance on natural data starts exactly from the pre-trained model's state and is only subtly adjusted during fine-tuning. The forward propagation for a clean example x is:

where the full pre-trained weight Wpre remains frozen. This design allows for effective adaptation to the downstream task while mitigating the risk of overfitting.

Crucially, this natural branch is auxiliary and is used exclusively during the training phase to provide a stable target for the adversarial branch. At inference time, matrices A and B are completely removed. This design choice not only enhances computational efficiency but also improves generalization by resulting in a more streamlined final model, a finding confirmed by our ablation studies in the appendix.

3.3 The Proposed PILO

The complete PILO framework, illustrated in Figure 2, integrates the two branches into a comprehensive multi-objective loss function:

where θnat = {Wpre + BA,Wfc} and θadv = {Wres + BadvAadv,Wfc}, and Wfc indicates the classifier parameters in the last fully connected layer. The classifier parameters Wfc are updated during training, while the pre-trained weights within θnat and θadv remain frozen.

This loss function has three components: (1) a standard cross-entropy loss on clean samples (x, y) to preserve accuracy; (2) a cross-entropy loss on adversarial samples (x∗ , y) to build robustness; and (3) a KL divergence term that regularizes the adversarial branch. This term encourages the model's output for an adversarial input x∗ to match the natural branch's output for the corresponding clean input x, promoting robust generalization. The coefficients λ1 and λ2 are dynamically scheduled based on validation accuracy

where e is the epoch, θ(e) represents the combined model parameters, and γ is a scaling hyperparameter. Acc(·) denotes the clean accuracy, and Dval denotes the validation set.

A primary advantage of PILO is its exceptional parameter efficiency. In contrast to traditional methods that fine-tune all m× n parameters of a weight matrix, PILO concentrates updates into four small, low-rank matrices (Aadv,Badv,A,B). This reduces the number of trainable parameters for each adapted layer to only r × (m + n) + radv × (m + n), a substantial reduction. This efficient parameterization offers multiple benefits: it significantly mitigates the risk of overfitting on limited data, enables better generalization, and reduces computational overhead during training. Furthermore, the removal of the auxiliary natural branch at inference results in a more streamlined final model, making PILO particularly well-suited for deployment on resource-constrained devices while achieving a superior balance between performance and robustness.

4 Experiments

In this section, we conduct a series of robustness benchmarks across multiple downstream tasks to evaluate the efficacy of our proposed PILO method against state-of-the-art approaches.

4.1 Experiment Settings

Downstream Datasets We evaluate PILO on five distinct datasets. CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009] are low-resolution image datasets, with CIFAR-10 comprising 50,000 training and 10,000 test images across 10 classes, and CIFAR-100 containing the same number of images distributed among 100 classes. For high-resolution benchmarks, we use Caltech-256 [Griffin et al., 2007], ensuring our data partitioning is consistent with that of AUTOLORA for fair comparison. For fine-grained classification, we use CUB-200 [Wah et al., 2011] (5,994 training and 5,794 test images across 200 bird species) and Stanford Dogs (DOG-120)[Khosla et al., 2011] (12,000 training and 8,580 test images of 120 dog breeds).

Pre-trained Models We use adversarially pre-trained ResNet architectures (ResNet-18 and ResNet-50) [He et al., 2016] as our primary backbones. These models were pretrained on the ImageNet dataset [Deng et al., 2009] with input images standardized to 224×224 pixels. Following the AUTOLORA configuration, the pre-trained models use a default adversarial budget of ϵpt = 4/255.

Training Configurations To ensure a fair comparison, we follow the training procedure of AUTOLORA, training all models for 60 epochs using SGD with a weight decay of 1 × 10-4 . For PILO, we set the rank of the auxiliary branch to r = 8. For the adversarial branch in ResNet models, we set the rank radv of the last two layers to be 4 times that of the first two layers to account for differing layer shapes. For Transformer-based models (ViT, Swin), we maintain a uniform radv across all layers. We hold out 5% of the training data as a validation set. Adversarial examples for training are generated using PGD-10 with ϵ = 8/255 and a step size of 2/255, using a batch size of 128. We adopt AUTOLORA's automated hyperparameter scheduling mechanism for λ1 and λ2, with γ = 6 by default. Detailed learning rates and specific radv values are provided in the appendix.

Evaluation We evaluate final model robustness against two standard adversarial attacks: Projected Gradient Descent (PGD-10) and AutoAttack (AA) [Croce and Hein, 2020b]. All attacks are constrained by the ℓ∞-norm with an adversarial budget of ϵ = 8/255. The PGD settings used for evaluation are identical to those used during training.

Baselines Our primary baselines for comparison are TRADES [Zhang et al., 2019], TWINS [Liu et al., 2023], and AUTOLORA [Xu et al., 2023]. To ensure a fair and direct comparison, we adopt the official hyperparameter configurations specified in their original papers, including learning rates, schedulers, weight decay, and any method-specific parameters (e.g., the β scalar in TWINS).

4.2 Experimental Results

As presented in Tables 2 and 3, our comprehensive benchmarks demonstrate that PILO consistently establishes a new state-of-the-art for robust fine-tuning. Across two different model architectures (ResNet18 and ResNet50) and five diverse downstream datasets, PILO significantly outperforms all three established baselines, i.e., TRADES, TWINS, and AUTOLORA, in adversarial robustness metrics under the attack of PGD-10 and AutoAttack. Crucially, these gains in robustness are achieved while maintaining or often improving accuracy on clean data, showcasing the effectiveness of our decoupled optimization strategy.

The results on the ResNet18 architecture, shown in Table 2, provide strong evidence for our central hypothesis. On challenging, fine-grained datasets with limited samples like CUB200 and DOG-120, where full-parameter models are highly susceptible to overfitting, PILO shows marked improvements in both robust and clean accuracy over AUTOLORA. This demonstrates PILO's ability to prevent the catastrophic forgetting of robustness; by performing a surgical update only on the principal components, PILO preserves the essential robust features learned during pre-training, while the auxiliary branch carefully adapts to the new data distribution. The consistent gains across all datasets validate that our method's implicit spectral regularization is more effective at generalizing on limited data than the full-parameter approaches of the baselines.

As exhibited in Table 3, the performance advantages of PILO are even more pronounced on the deeper ResNet50 architecture. For instance, on Caltech-256, PILO improves clean accuracy by a significant +2.79% over the best baseline, while also improving robustness. On CIFAR-100, it achieves a remarkable +1.72% gain in PGD-10 robustness and +1.24% in AA robustness over AUTOLORA. This trend suggests that as model capacity increases, the risk of disturbing the delicately learned features of a robust pre-trained model via full fine-tuning also increases. PILO's principled decomposition becomes even more critical in this high-capacity regime. It effectively isolates and preserves the most critical feature directions (the principal components) from a much larger and more complex parameter space, leading to superior scalability and performance.

The collective results empirically validate the core design principles of PILO. The consistent performance improvements are not coincidental but are a direct result of our targeted, spectrally-guided optimization strategy. By identifying and exclusively fine-tuning the parameter subspace most relevant to adversarial vulnerability, PILO achieves a more efficient and effective optimization than prior methods. These findings firmly establish PILO as a novel and state-of-the-art method for robust transfer learning.

4.3 Analysis and Ablation Studies

Impact of Adversarial Rank (radv) Table 4 presents our ablation on the adversarial rank radv, providing direct empirical support for our core hypothesis. The results reveal a clear, non-monotonic relationship between rank and performance, highlighting a crucial trade-off. A low rank (e.g., 18) provides insufficient capacity to capture the full complexity of the vulnerable subspace, limiting performance. Conversely, a high rank (e.g., 22) begins to degrade performance, which we attribute to robust overfitting; by including too many minor components, the model starts fitting to noise in the limited downstream data, mirroring the failure mode of fullparameter fine-tuning.

The optimal performance achieved at a moderate rank (e.g., radv = 19 for CUB-200, radv = 20 for CIFAR-100) represents the "sweet spot" that best isolates the essential vulnerable subspace. As shown in Table 6, these optimal ranks correspond to using only 30-40% of the original parameters, demonstrating that PILO's superior performance is intrinsically linked to its targeted parameter efficiency.

Generalization to Transformer Architectures To verify that our approach is not limited to CNNs, we evaluated PILO on Vision Transformer (ViT) and Swin Transformer backbones. The results in Table 5 show that PILO consistently improves adversarial robustness over AUTOLORA while maintaining comparable clean accuracy. This successful application to fundamentally different architectures suggests that our core principle, that adversarial vulnerability is concentrated in the principal components of linear transformation layers, is a general phenomenon in deep neural networks. It confirms that PILO is not an architecture-specific technique but a broadly applicable and robust fine-tuning paradigm.

Performance Across Different Pre-training Strengths We also investigated whether PILO's effectiveness depends on the initial robustness of the pre-trained model. Table 7 compares various fine-tuning methods on ResNet18 models pre-trained with different adversarial budgets (ϵpt). The results show two key trends: first, as expected, a stronger pretraining budget provides a better foundation and improves performance for all methods. Second, and more importantly, PILO consistently outperforms all baselines across every pretraining level. This demonstrates that PILO is not merely exploiting an artifact of a specific checkpoint; rather, it is a superior fine-tuning method that more effectively preserves and enhances the model's intrinsic robustness, regardless of its starting point.

5 Conclusion

This paper introduced PILO, a parameter-efficient framework for robust transfer learning designed to address the critical issue of catastrophic forgetting of robustness on limited downstream data. Our work is founded on the principle that adversarial vulnerability is disproportionately concentrated in the principal components of a pre-trained model's weights. By operationalizing this insight, PILO employs a principled, asymmetric decomposition strategy: it performs a surgical, spectrally-guided update on the principal components to build robustness, while a separate, auxiliary low-rank branch handles adaptation to natural data. This approach serves as a powerful form of implicit spectral regularization, effectively mitigating the gradient conflicts that plague monolithic finetuning methods. Our extensive experiments show that PILO consistently establishes a new state-of-the-art, outperforming previous approaches in adversarial robustness while using as little as 29-40% of the trainable parameters, thereby enhancing both model resilience and computational efficiency.

Limitations and Future Work We identify two primary limitations that suggest avenues for future work. First, the reliance on singular value decomposition (SVD) as a preprocessing step, while feasible for current models, may pose computational challenges when scaling to extremely large foundation models. Future research could explore more efficient algorithms for approximating the principal subspace. Second, the optimal adversarial rank, radv, is currently determined via hyperparameter search. Developing methods to automatically or dynamically select this rank would improve the framework's efficiency and ease of use, representing another valuable direction for future work.

Acknowledgments

This work was supported in part by the Chengdu Science and Technology Program under Grant 2025-YF12-00009-RC, in part by the Sichuan Science and Technology Program under Grant 2024ZYD0147, Natural Science Foundation of Xinjiang Uygur Autonomous Region under Grant 2024D01A18, the National Natural Science Foundation of China (NSFC) under Grant 62376228, in part by the Natural Science Foundation of Sichuan Province under Grant 2026NSFSC1468, and in part by the Open Project Program of the Network and Data Security Key Laboratory of Sichuan Province under Grant 2026NDSF002.