Proactive Federated Unlearning: Sensitivity-Guided Sparse Adaptation on Key Layers

1 Introduction

With increasing privacy requirements, Federated Learning (FL) enables multiple clients to collaboratively train a global model while keeping data local, making it attractive in domains such as healthcare, finance, and mobile services [McMahan et al., 2017; Ren et al., 2024]. Meanwhile, regulations such as the General Data Protection Regulation

• (b) The cost of forgetting.

Figure 1: Toy experiments on CIFAR-100. (a) Layer-wise sensitivity Sl measured at different training epochs, illustrating non-uniform sensitivity across layers and its increasing trend in deeper layers. (b) Forgetting error D(K) versus the number of modified layers K during post-hoc unlearning, showing that small/local modifications often fail to reach an acceptable forgetting threshold, while stronger forgetting typically requires modifying more layers.

(GDPR) and the "right to be forgotten" motivate the ability to remove the influence of specific clients, classes, or samples from an already trained federated model [Regulation, 2018]. This problem, known as Federated Unlearning (FU), aims to erase target data influence without full retraining [Wu et al., 2022b; Liu et al., 2022; Liu et al., 2024].

Most existing FUmethods are reactive, in the sense that unlearning is performed only after a request arrives [Liu et al., 2024; Zhao et al., 2025]. Prior work can be broadly grouped into several families. Update- or influence-based methods attempt to offset target contributions via gradient reversal, compensation, or approximation, which can be efficient but may rely on accurate update estimation and non-trivial correction [Halimi et al., 2022; Zhao et al., 2023; Khalil et al., 2025; Pan et al., 2025]. Retraining- or reconstruction-based methods more directly approximate a "never-learned" model, but are often expensive in federated settings with strict latency and resource constraints [Liu et al., 2021; Liu et al., 2022;

Che et al., 2023]. More recently, hybrid or structure-based designs, such as constrained parameterization and adapterstyle editing, have been explored to balance effectiveness and efficiency, although precisely localizing target knowledge can remain challenging under heterogeneous client distributions [Wu et al., 2022a; Zhong et al., 2025; Gao et al., 2022]. A practical mismatch therefore arises: unlearning requests typically occur after training, yet most training pipelines do not explicitly organize knowledge for selective removal, so post-hoc unlearning may require broader updates or additional overhead [Liu et al., 2024; Li et al., 2025].

To better understand the locality-cost trade-off in post-hoc unlearning, we conduct toy experiments on CIFAR-10/100 with a layer-wise sensitivity analysis. As shown in Figure 1(a), the sensitivity score Sl is highly non-uniform across layers and evolves over training: deeper layers become increasingly sensitive as optimization proceeds, while shallow layers remain relatively stable [Lee et al., 2019]. This trend suggests that later layers progressively capture more classand client-discriminative information, whereas earlier layers preserve more generic representations. Figure 1(b) shows that when updates are restricted to only a few layers (local correction), the forgetting error D(K) often stays above an acceptable threshold; reducing D(K) typically requires modifying a larger set of layers [Tarun et al., 2023]. This observation suggests that effective post-hoc forgetting under strict locality constraints can be difficult, and expanding the update scope increases computation and communication and may affect stability in long-running FL systems [Wu et al., 2022b; Gu et al., 2024].

Building on these observations, we focus on a budgeted setting where only a limited portion of the model can be modified during unlearning [Chowdhury et al., 2024; Li et al., 2024]. When the modification budget is small, post-hoc unlearning may not achieve sufficient forgetting; when the budget is expanded, the system cost increases. This motivates a proactive alternative: organize representations during training and track where unlearning-relevant information concentrates, so that future unlearning can be carried out within a small, well-identified subset of layers [Cao and Yang, 2015; Bourtoule et al., 2021; Li et al., 2025].

Motivated by this idea, we propose PFU-SKLA, a proactive FU framework guided by layer-wise sensitivity and sparse adaptation. PFU-SKLA has two phases. In the first phase, Proactive Federated Unlearning Readiness (PFUR), the model is trained with Orthogonality-guided Representation Decoupling (ORD) to reduce representation coupling [Ranasinghe et al., 2021], and the server periodically estimates layer-wise sensitivity using lightweight statistics. Here, sensitivity is a training-time signal of each layer's responsiveness to data and representation constraints; ORD does not define sensitivity, but can stabilize sensitivity estimates by reducing representation entanglement. Based on these signals, PFU-SKLA selects a compact set of critical layers via Dynamic Sensitivity-based Key Layer Identification (DS-KLI). In the second phase, when an unlearning request arrives, PFU-SKLA performs unlearning via sparse criticallayer adapters. The backbone parameters are frozen, and only lightweight, sparsely parameterized adapters are inserted into the selected critical layers and optimized using the retained data in a federated manner. Restricting updates to a small number of sparse parameters enables efficient and localized unlearning, while allowing exact rollback in parameter space by removing the adapters. The remaining layers stay untouched, helping preserve performance on retained data.

In summary, our main contributions are as follows:

• We propose PFU-SKLA, a proactive federated unlearning (FU) framework that prepares models during training for efficient, localized, and controllable unlearning at deployment.
• We develop two key components: (i) ORD to reduce representation coupling and sharpen unlearning-relevant signals, and (ii) DS-KLI to select a compact set of critical layers, enabling sparse adapter-based updates with exact rollback by adapter removal.
• Extensive experiments on FashionMNIST and CIFAR-10/100 with CNN and ViT backbones, across client/class/sample unlearning and IID/non-IID settings, demonstrate competitive retention-forgetting trade-offs while substantially reducing computation and communication compared with representative baselines.

2 RelatedWork

2.1 Machine Unlearning

Machine Unlearning (MU) aims to remove the influence of specific training data from a trained model without full retraining, in order to meet privacy, compliance, and fairness requirements [Bourtoule et al., 2021; Koundinya Gundavarapu et al., 2024; Li and Sui, 2025; Liu et al., 2025; Zhang et al., 2023]. In centralized settings, existing MU approaches can be broadly categorized into data-level and model-level methods [Mercuri et al., 2022; Nguyen et al., 2025; Li et al., 2025]. Data-level methods operate by modifying or reorganizing the training data, such as through data mixing, substitution, or partial retraining [Zhang et al., 2022; Graves et al., 2021; Tarun et al., 2023; Eisenhofer et al., 2025]. While computationally efficient, these approaches offer limited control over where and how knowledge is removed, and do not naturally support built-in forgettability or reversibility, making them less suitable for federated and proactive unlearning scenarios. Model-level methods directly manipulate model parameters via pruning, parameter correction, or sub-model replacement to erase target knowledge [Chowdhury et al., 2024; Li et al., 2024; Warnecke et al., 2021]. Although more precise, they are typically reactive, may incur high computational costs, and risk catastrophic forgetting when large parameter subsets are modified. Several theoretically grounded frameworks, such as PAC-Bayesian variational unlearning and cryptographic verification, further improve unlearning guarantees [Jose and Simeone, 2021; Eisenhofer et al., 2025], but are mainly designed for centralized settings.

2.2 Federated Unlearning

Federated Unlearning (FU) focuses on removing the influence of specific clients, classes, or samples from a fed-

Training Stage

(PFUR: Proactive Federated Unlearning Readiness)

Unlearning Stage

(FU-SKLA: Federated Unlearning via Sparse Critical-Layer Adapters)

erated model while preserving its utility on the remaining data [Liu et al., 2021; Liu et al., 2022; Su and Li, 2023; Wu et al., 2022a; Halimi et al., 2022; Zhao et al., 2023; Khalil et al., 2025; Deng et al., 2024; Pan et al., 2025]. Compared with centralized unlearning, FU introduces additional challenges due to distributed data ownership, limited client participation, and strict constraints on communication and computation [Liu et al., 2024; Zhao et al., 2025]. As a result, existing FU methods often face a trade-off between unlearning effectiveness and system efficiency.

Passive FederatedUnlearning. Most existing FUmethods adopt a passive paradigm, where unlearning is triggered only after a request is issued. These approaches are well motivated in practice, as unlearning requests are typically sporadic and unpredictable, and retraining from scratch is often infeasible. Representative methods include FedEraser [Liu et al., 2021], which leverages stored historical updates for efficient model reconstruction, FUKD [Wu et al., 2022a], which removes client influence via update subtraction and knowledge distillation, and NoT [Khalil et al., 2025] and FedOSD [Pan et al., 2025], which mitigate forgetting and optimization conflicts through parameter manipulation and optimization strategies. Despite their effectiveness, passive FU methods generally require either broad parameter updates or access to auxiliary server-side information, leading to high communication and computation costs and limited controllability. Moreover, since no structural preparation is made during training, these methods often struggle to localize and erase target knowledge without affecting retained information, especially in long-running and dynamic FL systems.

Reversible Federated Unlearning. Recent efforts have explored reversible FU to mitigate irreversible forgetting. The FUSED method introduces sparse adapters to overwrite existing knowledge, enabling rollback by removing the adapters [Zhong et al., 2025]. While promising, FUSED identifies critical layers primarily based on global parameter variations, which may fail to capture locally sensitive or client-specific knowledge distributions in heterogeneous federated settings. This can lead to less precise unlearning and unnecessary updates to non-critical layers.

3 Methodology

As illustrated in Figure 2, PFU-SKLA adopts a two-stage design that separates unlearning preparation from unlearning execution. This separation is essential because effective FU requires sensitive knowledge to be structurally identifiable during training, rather than relying on costly post-hoc corrections. In the first stage, termed Proactive Federated Unlearning Readiness (PFUR), the model is trained to disentangle representations and track layer-wise sensitivity signals, enabling the localization of class- and client-specific information into a compact set of critical layers. In the second stage, unlearning is carried out by attaching sparse adapters only to these critical layers while freezing all original parameters, ensuring that updates remain lightweight, localized, and fully reversible by removing the adapters, with minimal interference to retained knowledge. Notably, proactive refers exclusively to training-time structural or-ganization, not preemptive erasure.

3.1 Proactive Federated Unlearning Readiness

PFUR consists of two core components: orthogonalityguided representation decoupling (ORD) and periodic critical-layer updates (PCU).

Orthogonality-Guided Representation Decoupling

To facilitate more reliable identification of layers that support representation decoupling, we introduce ORD during training. By reducing representation entanglement, the orthogonality-guided (OG) constraint sharpens layer-wise sensitivity signals and leads to more stable critical-layer identification.

Assume client i with dataset Di = {(xi,j, yi,j)}|Di| j=1 and local model parameter wi = (ϕi, vi), where ϕi is the feature extractor and vi is the classifier. We utilize phii to get feature representations fi = {fi,j}|Di| j=1 = {fθi (xi,j)}|Di| j=1 . Then we perform ℓ2 normalization for fi as follows:

and construct a client-local similarity matrix over normalized features for computing the OG regularization:

Let Pi = {(j, k) | yi,j = yi,k, j ̸= k} denote the sets of positive pairs and Ni = {(j, k) | yi,j ̸= yi,k} denote the sets of negative pairs, respectively. We define the mean similarities as follows:

where si,pos is the positive similarity and si,neg is the negative similarity. Then we get the OG constraint as follows:

where γ > 0 balances the penalties on positive and negative pairs. Minimizing LOG drives same-class features to be tightly clustered (spos → 1) and different-class features to be nearly orthogonal (sneg → 0), thereby enforcing explicit inter-class separation and intra-class compactness in the feature space. We integrate this OG regularizer with the task objective Li,CE as follows:

where λ controls the strength of the OPL regularization. A single local gradient descent step at iteration t is given by

where ηt is the local learning rate. These updated local models are aggregated on the server to get wt via FedAvg.

Through its OG constraint, ORD reshapes the representation space during federated training, reducing cross-class interference while guiding the periodic critical-layer updates.

Periodic Critical-Layer Updates

During federated training, the server periodically updates the critical-layer set using layer-wise parameter update magnitudes and aggregated norms of the OG gradients, which are computed locally by clients and communicated only as lightweight statistics. Specifically, for each layer l, the parameter update and OG gradient norm are as follows:

We combine the two normalized measures using a convex combination to balance update magnitude and representation sensitivity, resulting in a sensitivity score that is empirically more stable than relying on a single signal:

In addition, we utilize an exponential moving average (EMA) to update the critical-layer score as follows:

The server performs this procedure every R rounds to periodically update the layer relevance scores. After training, the top-K layers with the highest scores are selected as critical layers. Moreover, given the classifier's sensitivity to heterogeneous data distributions, we always include it as a critical layer. The selection of the critical layer set Lf is as follows:

where top-K(PCl) is the index of top-K layer and -1 is the classifier.

3.2 Federated Unlearning via Sparse Critical-Layer Adapters

After PFUR identifies the critical layer set Lf , PFU-SKLA performs unlearning by freezing the global model and attaching sparse adapters to these layers. Although PFUSKLA adopts an adapter-based mechanism similar to reversible methods such as FUSED, it differs in that the adapted layers are proactively identified during training based on representation-level sensitivity rather than selected post hoc from global parameter variations.

Specifically, for each critical layer l ∈ Lf , the server initializes an adapter Al whose shape matches that of the original global layer weights wl. All global parameters {wl} are frozen and remain unchanged during unlearning, while only the adapter parameters are trainable. We adopt element-wise sparsity for adapters. For each l ∈ Lf , we associate a binary mask Ml ∈ {0, 1}shape(wl) with P(Ml,j = 1) = p. Only the entries with Ml,j = 1 are allowed to be updated; the remaining entries are fixed to zero throughout unlearning. (Equivalently, one may parameterize Al = Ml ⊙ ˜ Al and optimize ˜ Al .)

During the unlearning stage, the server distributes the current adapters {Atl }l∈Lf (and masks {Ml}l∈Lf , if needed) to the participating clients in the retained set Cremain. Each client i reconstructs the effective model using the frozen base weights {wl} and updates only the adapter parameters by minimizing the loss on its retained data Di,remain:

where LCE preserves performance on the retained data, and LOG is reused to maintain the representation geometry established during training, stabilizing adapter updates rather than introducing additional regularization. Client i performs a gradient descent step to update the sparse adapter parameters:

where ⊙ denotes element-wise multiplication.

After local updates, the server aggregates the adapters from participating clients using FedAvg:

where ni is the retained data size of client i and Nremain = ∑i∈Cremain ni.

For client unlearning, only clients in Cremain participate and each client trains on its retained data. For class unlearning or sample unlearning, the procedure is identical except that the target classes or samples to be forgotten are excluded from Di,remain, and only the remaining data are used for local updates.

After Tu unlearning rounds, the final model is constructed as:

The forgetting effect is achieved solely through the learned sparse adapters {ATu l }. As the global base parameters remain unchanged, PFU-SKLA is reversible:

The original model can be recovered by removing the adapters, enabling controllable and lightweight FU.

3.3 PFU-SKLAOverview

PFU-SKLA integrates PFUR and FU-SKLA into a unified unlearning pipeline. PFUR proactively prepares the model by decoupling representations and identifying key layers, while FU-SKLA updates only these layers using sparse adapters during unlearning. This yields efficient, localized, and reversible unlearning with minimal communication and computation. The procedure is summarized in Algorithm 1.

4 Theoretical Analysis

This section provides a concise theoretical justification for the efficiency and effectiveness of PFU-SKLA. We first show that ORD in PFUR encourages decorrelated functional responses, and then explain why FU-SKLA yields low interference and exact reversibility. In essence, ORD reduces the overlap between gradient subspaces of retained and target knowledge, enabling sparse adapters to remove target knowledge with limited collateral damage.

Algorithm 1: PFU-SKLA Algorithm

Input: Initial model w, training rounds T, unlearning rounds Tu, client set C, remain-set clients Cremain, hyperparameters β, R, λ, α, sparsity rate p Output: Unlearned model wfinal 1 Stage 1: PFUR; 2 Initialize PCl and Lf ; 3 for t = 1 to T do 4 for i ∈ Ct do 5 perform local ORD training using Eq. 6 and upload wt+1 i ; 6 Update wt+1 via FedAvg; 7 if t mod R = 0 then 8 Update PCl and Lf using Eq. 10 and 11; 9 Stage 2: FU-SKLA; 10 Freeze the global model w; 11 Initialize masks Ml with P(Ml,j = 1) = p and adapters Al = 0 for l ∈ Lf ; 12 for t = 1 to Tu do 13 for i ∈ Cremain do 14 update Ai,l using Eq. 13 and upload {Ai,l}l∈Lf ; 15 Server aggregates Al ← FedAvg({Ai,l}) for l ∈ Lf ; 16 Construct wfinal using Eq. 15; 17 return wfinal ;

4.1 Foundations of Proactive Unlearning Readiness

We adopt a functional mapping view for PFUR. Let Wl ∈ RCout×D denote the flattened kernels of layer l, where each row wl,i is a filter vector and Cout ≤ D. Each filter induces a linear functional Fl,i(x) = ⟨wl,i, x⟩ acting on a normalized local input x ∈ RD . ORD applies the orthogonality-guided regularizer LOG = ∥WlW ⊤ l - I∥2F .

Lemma 1 (Approximate Orthonormality of Filters). If LOG → 0, then for any filters wl,i,wl,j, we have ⟨wl,i, wl,j ⟩ ≈ δij .

Theorem 1 (Functional Decorrelation under Near-Isotropic Inputs). Assume the normalized local input has covariance Σ satisfying ∥Σ- I∥2 ≤ δ. Given Lemma 1, for all i ̸= j,

In particular, when LOG is small (so |w⊤ l,iwl,j| is small), the cross-correlation is bounded by a small quantity.

Theorem 1 shows that ORD reduces cross-coupling among functional responses. This alone does not guarantee clean separation between target and retained knowledge, but offers a useful mechanism: if target- and retained-related information concentrates on different feature subsets, weaker crosscorrelation reduces interference between them and can facilitate subsequent localization and selective editing.

Remark (Reduced Gradient Interference). By weakening coupling between target- and retained-dominated components, ORD makes the gradients induced by the retained objective T1 and unlearning objective T2 less aligned in expectation. This reduces overlap between their effective update subspaces and favors sparse, localized unlearning updates.

4.2 Stability and Reversibility of FU-SKLA

FU-SKLA freezes base parameters Θ(0) and updates only sparse critical-layer adapters a injected via a fixed operator P, i.e., Θ = Θ(0) + Pa. Under smoothness and local firstorder approximation, and with an element-wise adapter mask activated at rate p, the expected retained-loss change induced by optimizing the unlearning objective is

where η is the learning rate.

Eq. (17) captures the leading-order interference term for a single masked adapter step. Under standard smoothness and small-step conditions, the same cross-term accumulates along the unlearning trajectory, while higher-order terms are of order O(η2 ).

Controlling Higher-Order Collateral Damage. The retained-loss change also includes a higher-order remainder R(∆Θ). With ∆Θ = P ∆a (and sparsity), smoothness yields |R(∆Θ)| ≤ L2 ∥∆Θ∥22 . ORD encourages more concentrated, target-oriented updates, which reduces the magnitude of update components affecting retained knowledge and tightens this bound in expectation. Finally, since unlearning is implemented only through additive adapters while Θ(0) remains unchanged, the original model is exactly restored by removing the adapters, yielding Θrestored = Θ(0) .

5 Experiments

5.1 Experimental Setting

In this section, we briefly summarize our experimental evaluation on FashionMNIST and CIFAR-10/100 under federated learning with 20 clients and full participation. We compare PFU-SKLA against representative FU baselines and report results from effectiveness, privacy, and efficiency perspectives.

5.2 Experimental Analysis

Effectiveness in Knowledge Retention and Forgetting

As shown in Table 1, PFU-SKLA achieves strong retentionforgetting performance across client, class, and sample unlearning. It attains high RA with near-zero FA on FashionMNIST, competitive RA on CIFAR-10 with effective forgetting (FA=4.15 for client and FA=0.00 for class), and maintains stable retention on the more challenging CIFAR-100 while keeping FA low (e.g., FA=0.15 for client unlearning). For sample unlearning, PFU-SKLA consistently yields strong retained-set performance, achieving 0A/PS of 100.00/100.00 on FashionMNIST, 71.66/76.47 on CIFAR-10, and competitive results on CIFAR-100 (e.g., 0A=75.00 on ResNet-18 and 0A/PS=100.00/70.00 on SimpleViT).

Retrain

PFU-SKLA

1

3

5

10

Client Number

Figure 3: Impact of the number of forgotten clients on RA and FA for Retrain and PFU-SKLA on the CIFAR-10 with IID setting.

Computational and Communication Efficiency

PFU-SKLA consistently achieves the lowest computation and communication costs by updating only sparse critical-layer adapters. On CIFAR-10 and CIFAR-100, it substantially reduces Comp and communication compared with FedOSD and FUSED, and it also achieves very low communication overhead on FashionMNIST (Table 1).

Privacy Preservation and Relearning Accuracy

PFU-SKLA shows favorable privacy behavior, with relatively low MIA in client unlearning and low relearning accuracy in challenging settings, suggesting reduced membership leakage and limited recoverability of forgotten knowledge (Table 1).

Evaluating Knowledge Isolation and Retention

To study unlearning effects on unique and overlapping knowledge, we evaluate CIFAR-10 client unlearning in Table 2. The forgotten client removes all class-1 and 90% of class-0, while the remaining clients keep classes 2-9 and the remaining 10% of class-0. All methods achieve UF-Acc=0.00. PFU-SKLA yields the best O-Acc (19.81) and UR-Acc (60.48), indicating better retention of both overlapping (class-0) and unique retained knowledge (classes 2-9).

Performance of Multi-Client Unlearning

As shown in Figure 3, when the number of forgotten clients increases from 1 to 10 (50% of total clients), both Retrain and PFU-SKLA show a gradual decline in RA and a slight increase in FA. Specifically, PFU-SKLA maintains comparable RA to Retrain across all settings while achieving effective forgetting with only minor FA differences. Although Retrain attains marginally lower FA, it comes with much higher computational and communication overhead. Overall, the results demonstrate that PFU-SKLA achieves a balanced trade-off between retention and forgetting, maintaining stable performance as the number of forgotten clients grows while significantly improving unlearning efficiency.

Ablation Study

The ablation results in Table 3 support the contribution of ORD and our sensitivity-based key-layer identification (DSKLI). Removing ORD (w/o ORD) consistently reduces retention and privacy-related metrics across datasets, suggesting that ORD prepares more disentangled representations for stable unlearning. We also compare key-layer selection strategies during unlearning. Relative to sensitivity-based selection, the K random layers strategy leads to clear performance drops, especially for sample unlearning, where both

FA(%)

6

4

2

0

0A and PS decrease substantially on CIFAR-10/100 and SimpleViT, indicating stronger interference with retained knowledge. The Last-K layers strategy improves over random selection in some cases but remains consistently below PFUSKLA, suggesting that a fixed heuristic may not reliably locate layers most responsible for target knowledge. Overall, DS-KLI provides a more consistent key-layer choice than random or last-layer heuristics under the same adapter budget, leading to improved retention-forgetting trade-offs.

6 Conclusion

In this work, we propose PFU-SKLA, a proactive, efficient FU framework. It introduces proactive unlearning readiness during training via ORD-based representation decoupling and sensitivity-based key-layer identification, enabling targeted, lightweight deployment-time unlearning. It supports exact reversibility because unlearning uses additive sparse adapters while the base model remains frozen, so removing them restores the pre-unlearning model. Experiments on FashionMNIST and CIFAR-10/100 show PFU-SKLA achieves competitive retention-forgetting trade-offs while reducing computation and communication versus representative baselines.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62271128, U2333207); the Major Science and Technology Project of Sichuan Province (2025ZDZX0093); the Science and Technology Program of Sichuan Province (2022ZDZX0004, 2025ZHRG0006); the "Challenge-Based" Project of Sichuan Provincial Science and Technology Program (2023YFG0374); the Regional Innovation Cooperation Project of Sichuan Province (2025YFHZ0302); and the Key Research and Development Support Program of Chengdu (2025-YF08-00128-GX, 2025YF12-00029-RC, 2026-YF11-00032-HZ).