MOBO: A Merging-Oriented Bi-Level Optimization Framework for Class Incremental Learning

1 Introduction

In many open-world applications, data arrive sequentially and the set of classes expands over time, requiring learning systems to incrementally acquire new class knowledge without revisiting previous data [Thrun, 1995]. This paradigm, known as Class-Incremental Learning (CIL) [Rebuffi et al.,

∗Corresponding author.

2017], faces the fundamental challenge of catastrophic forgetting, where adaptation to new tasks disrupts representations learned for earlier classes. Recent advances in PreTrained Models (PTMs) [Wang et al., 2022c] have inspired PTM-based CIL methods, which freeze the pretrained backbone and optimize lightweight components-such as prompts or adapters-to adapt to incremental tasks. By leveraging the generalizable representations acquired from large-scale data, PTM-based CIL approaches have demonstrated superior performance compared to models trained from scratch [Zhou et al., 2024b; Zhou et al., 2024c]. However, maintaining an ever-growing set of task-specific adapters introduces significant scalability challenges in terms of memory and deployment, especially over long task sequences [Sun et al., 2025].

To mitigate this overhead, merging-based pre-trained CIL has emerged as a promising direction. Instead of maintaining separate task-specific modules at inference time, these methods consolidate them into a single model by merging parameters or adapter weights, achieving competitive accuracy with high inference efficiency. For instance, ACMap [Fukuda et al., 2025] utilizes a simple average-based merging strategy to curb parameter growth while preserving performance, while PM [Qiu et al., 2025] investigates the computation of optimal merging coefficients via a closed-form solution. Despite their effectiveness, most merging-based methods follow a decoupled pipeline: they first train a task-specific adapter optimized primarily for the current task, then post hoc seek optimal coefficients to merge it into the previously learned model.

A key limitation of this decoupling is that successful merging depends not only on the merging coefficients but also on the compatibility among the models being merged. When task-specific adapters are trained independently, their parameters can drift into divergent regions of the solution space. Such incompatibility can induce severe conflicts during integration, leading to substantial performance degradation and exacerbated forgetting. As illustrated in Figure 1, this paradigm (represented by ACMap) may cause the new task model's loss on previous classes to increase rapidly, ultimately harming the final merged model. These observations motivate a natural question: Can we explicitly optimize the new task parameters to be "merge-friendly" during the training phase?

To address this issue, we propose a merging-oriented bilevel optimization framework for class-incremental learning (MOBO). The core idea is to synergize the learning of new tasks with the preservation of the global merged model. We formulate the problem as a bi-level optimization process. At the upper level, we introduce a merging-oriented global loss that anticipates the merged model's performance and guides the optimization of current task parameters toward mergefriendly solutions. At the lower level, we adopt a dynamic weighted merging strategy that optimizes merging coefficients and updates the merged model. By alternating between these two stages, MOBO jointly encourages effective learning on the new task and compatibility with the previously learned model, thus mitigating catastrophic forgetting caused by conflicting integrations.

The main contributions of this paper are summarized as follows:

• We propose MOBO, a bi-level optimization framework for CIL that synergizes the training of task-specific adapters with the integration of the merged model, effectively resolving the incompatibility issues inherent in decoupled approaches.
• We introduce a merging-oriented global loss tailored for the exemplar-free setting. By decomposing the objective into manageable sub-losses, MOBO guides the optimization trajectory toward a "merge-friendly" region without requiring access to historical data.
• Extensive experiments on CIFAR-100, CUB-200, ImageNet-R, ImageNet-A, and VTAB demonstrate that MOBO outperforms state-of-the-art methods and exhibits superior robustness and scalability, particularly in scenarios with long task sequences.

2 RelatedWork

Class-incremental learning (CIL) aims to enable models to continuously learn from new task data while preserving performance on previously learned tasks [Ratcliff, 1990]. The primary challenge in CIL is catastrophic forgetting, where the adaptation to new tasks degrades previously acquired knowledge. Existing approaches typically fall into four representative paradigms. Replay-based methods (e.g., [Aljundi et al.,

2019b; Chaudhry et al., 2019]) mitigate forgetting by storing and replaying a subset of samples from previous tasks, thereby recovering historical knowledge during training [Liu et al., 2020]. Knowledge distillation methods (e.g., [Dhar et al., 2019; Douillard et al., 2020]) transfer knowledge by aligning the representations of the updated model with those of the old model [Li and Hoiem, 2017; Rebuffi et al., 2017]. Parameter regularization methods (e.g., [Aljundi et al., 2018; Aljundi et al., 2019a]) penalize changes to parameters critical for prior tasks, thus limiting parameter drift [Kirkpatrick et al., 2017]. Finally, model correction methods (e.g., [Belouadah and Popescu, 2019; Pham et al., 2022]) aim to rectify the prediction bias towards new classes to ensure balanced learning [Pian et al., 2023]. Despite their effectiveness, these approaches often rely on assumptions that are difficult to satisfy in real-world scenarios, such as access to raw historical data or significant storage and computational resources.

Recently, CIL based on pre-trained models (PTMs) has emerged as a prominent research direction. In this setting, parameter-efficient fine-tuning (PEFT) methods [Han et al., 2024] are widely adopted to mitigate catastrophic forgetting while preserving the robust representational capacity of PTMs. Prompt-based methods guide the PTM's behavior via learnable prompts; for instance, L2P [Wang et al., 2022c] utilizes a prompt pool with key-query matching, while DualPrompt [Wang et al., 2022b] incorporates global and expert prompts to encode task-shared and task-specific knowledge. Adapter-based methods introduce lightweight modules for adaptation, such as EASE [Zhou et al., 2024c], which employs dynamic subspace expansion and prototype synthesis. MOS [Sun et al., 2025] further optimize the training strategy for task-specific adapters by leveraging prior knowledge from the pretrained model to enhance performance. To further optimize inference efficiency, merging-based strategies have recently been proposed to consolidate task-specific modules into a single model. ACMap [Fukuda et al., 2025] utilizes a simple average-based merging strategy to curb parameter growth while preserving performance. PM [Qiu et al., 2025] extends this direction by investigating the computation of optimal merging coefficients via a closed-form solution. However, these methods typically follow a decoupled pipeline-training task-specific adapters first and merging them post-hoc-which often overlooks the parameter compatibility required for effective integration.

3 Preliminaries

3.1 Problem Setting

Consider a sequence of classification tasks with disjoint datasets {Dt}Tt=1 , where each Dt = {(Xt, Yt)} consists of input-label pairs. The label spaces are non-overlapping across tasks, i.e., Yt ∩ Yt′ = ∅ for all t ̸= t′ . Following the standard CIL protocol, once training on task t is completed, data from previous tasks are no longer accessible. Moreover, at inference time, the task identity of a query is unknown (task-ID free); the model is required to classify samples over the union of all classes ⋃Tt=1 Yt.

3.2 Pre-TrainedModel Based CIL

Recent state-of-the-art CIL approaches are predominantly built upon pre-trained models. They typically leverage a frozen pre-trained backbone as the encoder, along with trainable, randomly initialized linear classifiers [Wang et al., 2022c]. Formally, let F(·;Φ) : RD → Rd denote the pretrained encoder with frozen parameters Φ, and let W ∈ Rd×|Y| denote the trainable classifier weights. For an input query x, the prediction is given by

Beyond training only the classifier, some methods further leverage parameter-efficient fine-tuning modules, such as adapters and prompts. A standard adapter follows a bottleneck architecture consisting of a down-projection Wdown ∈ Rd×r and an up-projection Wup ∈ Rr×d , where r ≪ d is the bottleneck dimension [Zhou et al., 2024c]. Given an input feature xin, the adapter output is injected via a residual connection to the MLP layer:

where σ(·) denotes a non-linear activation function. Adapters are typically inserted into the MLP sublayer of each of the Nblocks transformer blocks.

To alleviate the significant parameter overhead introduced by maintaining separate adapters for every task, several recent methods adopt adapter merging strategies. The core idea is to maintain a single consolidated adapter model. As exemplified by ACMap [Fukuda et al., 2025], after learning the adapter parameters θt for the new task, the method merges them into the global parameters ¯ θ via an exponential moving average:

The overall performance of the final merged model is jointly determined by the compatibility of the adapter parameters and the selection of merging coefficients [Fukuda et al., 2025; Qiu et al., 2025].

4 Method

4.1 Overview

While merging-based methods have demonstrated competitive performance, they often suffer from significant catastrophic forgetting, as illustrated in Figure 1. A key reason for this limitation is that these approaches optimize each taskspecific adapter θt solely to minimize the loss on the current task (cf. Figure 2(a)), without explicitly considering the performance of the merged model. As a result, naively aggregating incompatible adapter parameters leads to substantial accuracy degradation and exacerbates forgetting.

In contrast, we propose to optimize θt to directly maximize the performance of the merged model across all previously seen tasks. Formally, we frame this as a bi-level optimization problem (cf. Figure 2(b)):

Here, ¯ θt-1 denotes the merged adapter obtained after task t - 1, and ϕ(α) represents the objective function that determines the optimal merging coefficient (see Eq. (14) for details). In this formulation, the upper level optimizes the new task parameters θt to minimize the loss of the merged model, while the lower level determines the optimal merging coefficient α⋆ that governs the aggregation of the previous and current models.

4.2 Upper Level Optimization

According to Eq. (4), the upper-level objective aims to optimize θt to minimize the cumulative loss of the merged model

¯ θt over all observed data X1:t. However, directly implementing it is infeasible due to the CIL constraint that the data from previous tasks (i.e., X1:t-1) is no longer accessible.

To address this issue, we reformulate the optimization objective and derive the following decomposed form:

where the discrepancy terms are defined as follows:

Note that Lce(¯ θt-1,X1:t-1) is constant with respect to θt and therefore does not influence the optimization. Consequently, the joint objective decomposes into three interpretable components: Lce(θt, Xt) optimizes the new task model θt on current data; ˙∆t encourages consistency between the merged model ¯ θt and the new task model θt; and ¨∆t promotes compatibility between the merged model ¯ θt and the previous merged model ¯ θt-1. This decomposition transforms the intractable global objective into manageable sub-objectives, which can be approximately solved as described below.

Furthermore, following [Fukuda et al., 2025], we decouple the optimization of the feature extractor and the classifier to better balance stability and plasticity. Specifically, we reformulate the discrepancy term ˙∆t in Eq. (6) as:

Here, the superscripts F and C denote the feature extractor and classifier, respectively. A similar decomposition is applied to ¨∆t, yielding the sub-terms ¨∆Ct and ¨∆Ft .

Feature extractor optimization. The discrepancies in the loss terms are primarily attributed to variations in the feature representations, given that the classifier is fixed:

However, since the previous data X1:t-1 is unavailable in Eq. (8), we approximate ¨∆Ft using a first-order Taylor expansion around ¯ θF t-1 :

where the accumulated gradients ∑t-1 i=1 ∇θF i F(Xi; θF i ) are maintained throughout training. Consequently, the overall optimization objective for the feature extractor is defined as:

To further enhance discriminability between new and previously learned tasks in the merged model's feature space, we introduce a prototype-based loss:

where cos(·, ·) denotes cosine similarity and P t-1 1:t-1 represents the set of class prototypes from all prior tasks. By penalizing high similarity between the feature representations of the current task and the prototypes of previous tasks, this loss encourages greater separation in the feature space, thereby improving inter-class discriminability.

Classifier optimization. Since MOBO employs a prototype-based classifier, optimization primarily involves updating these prototypes. For the discrepancy ˙∆Ct between the current and merged models, the availability of current task data enables direct computation and integration of the current task's prototype into the merged model. Conversely, for ¨∆Ct , where previous data is inaccessible, we approximate the prototypes of prior tasks in the merged model via prototype alignment. Additionally, to account for shifts in feature distribution introduced by new tasks, we incorporate an incremental covariance update. Specifically, the update mechanism is defined as:

where P i j denotes the prototype of task j within the i-th merged model, Σi is the task-specific feature covariance, and Ni is the sample size for task i. As illustrated in Figure 2 (c), these incremental updates to class prototypes and covariance matrices enable consistent modeling of class distributions across tasks. Leveraging these statistics, we employ a Mahalanobis-distance-based classifier during inference [Goswami et al., 2023], thereby facilitating unified discrimination between old and new tasks.

Global loss. Building upon the preceding analysis, we formulate the global loss function as follows:

where λ1 and λ2 are hyperparameters balancing the contributions of each loss component. The global objective Lglobal consists of three terms: Lce, which ensures performance on the current task; Lf , which optimizes the quality of the merged model; and Lproto, which promotes class discriminability. Together, these components synergistically enhance generalization and model merging effectiveness.

4.3 Lower Level Optimization

As described in Eq. (4), the lower-level optimization seeks to determine the optimal merging coefficient α that enhances the merged model ¯ θt. Following the approach of [Qiu et al., 2025], we compute this coefficient by minimizing the performance degradation caused by the merging process. Formally, this objective is expressed as:

Algorithm 1 The optimization procedure of MOBO

Input: Dataset {(Xt, Yt)}Tt=1 , Iteration number N Output: Merged parameters ¯ θT 1: if t = 1 then 2: θ1 ← argminθ1 Lce(θ1,X1) 3: ¯ θ1 = θ1 4: end if 5: for t = 2 to T do 6: for iteration = 1 to N do 7: // The Upper Level, refer to Eq. (13) 8: θt ← argminθt Lglobal(θt, ¯ θt, Xt) 9: // The Lower Level, refer to Eq. (18) 10: qi = (¯ θt-1 - θt)⊤Hi(θi)(¯ θt-1 - θt) 11: α⋆ = ∑t-1 i=1 qi qt+∑t-1 i=1 qi 12: ¯ θt = α⋆¯ θt-1 + (1 - α⋆ )θt 13: end for 14: end for 15: Return ¯ θT

where Li(¯ θt) denotes a shorthand for Lce(¯ θt, Xi).

To streamline computation and improve efficiency, we approximate Li(¯ θt-1) ≈ Li(θi) for all i ≤ t - 1. Thus, the objective function for the merging coefficient becomes:

Next, we perform a second-order Taylor expansion of Li(¯ θt) around θi:

where Hi(θi) is the Hessian matrix of Li evaluated at θi. Since θi is approximately a local minimizer of Li, the gradient term vanishes (i.e., ∇Li(θi) ≈ 0). Under this assumption, Eq. (16) simplifies to:

Combining Eq.(15) and Eq.(17), we derive the analytical solution for the merging coefficient:

In practice, to reduce computational complexity, Hi(θi) is approximated and stored using the corresponding Fisher Information Matrix Fi(θi).

Finally, the overall MOBO procedure is summarized in Algorithm 1.

5 Experiments

5.1 Datasets for Evaluation

Datasets: We evaluate our method on five benchmark datasets: CIFAR-100 (CIFAR) [Krizhevsky and Hinton,

2009], CUB [Wah et al., 2011], ImageNetR (IN-R) [Hendrycks et al., 2021a], ImageNet-A (IN-A) [Hendrycks et al., 2021b], and VTAB [Zhai et al., 2019]. CIFAR-100 contains 100 classes, CUB, ImageNet-R, and ImageNet-A each contain 200 classes, and VTAB contains 50 classes. Following [Fukuda et al., 2025], for all datasets except VTAB, the class order for each seed is randomized. For VTAB, the class order is fixed. We use the notation "B-m Inc-n" to divide these datasets into T tasks, where m is the initial number of classes and n is the number of classes added incrementally for each task.

Evaluation Metrics: Following the standard protocol in CIL [Rebuffi et al., 2017], we use two evaluation metrics: the average accuracy ¯ A across all tasks and the final accuracy AT (final accuracy) for the final task.

Baselines: We compare our method with several baselines and state-of-the-art methods. The baseline method is finetuning the pre-trained model for each task, referred to as Finetune. We also test finetuning only the adapter, referred to as Finetune Adapter. For comparison, we select CIL approaches using a pre-trained model: L2P [Wang et al., 2022c], DualPrompt [Wang et al., 2022b], CodaPrompt [Smith et al., 2023], SimpleCIL [Zhou et al., 2024a], APER [Zhou et al., 2024a], EASE[Zhou et al., 2024c], ACMap[Fukuda et al., 2025], MOS[Sun et al., 2025], and NSDCIL[Wu et al., 2025]. Among these approaches, except for L2P , DualPrompt, and CodaPrompt, all other methods are adapter-based or lorabased solutions. Moreover, ACMap is our primary baseline for comparison, as it is the first work to consider an adaptermerging strategy for class-incremental learning.

Training Details: We follow the training configurations used in [Fukuda et al., 2025]. For the pre-trained model, we use the ViT-B/16 model, pre-trained on ImageNet-21K [Ridnik et al., 2021]. Adapter is optimized using SGD with cosine annealing for the learning rate scheduler. We tuned hyperparameters such as the learning rate, batch size, and training epochs for each dataset using the validation set, and report the final configurations in the supplementary materials.

5.2 Main Results

In this section, we compare our method with the state-of-theart CIL approaches based on pre-trained models. Table 1 reports the average accuracy ¯ A and final accuracy AT on the five benchmark datasets. In addition, Figure 3 illustrates the incremental performance trends of comparative methods. The experimental results demonstrate that our proposed method achieves strong performance in terms of both ¯ A and AT , particularly on challenging datasets with significant domain shifts. For instance, on ImageNet-A, compared with ACMap, which also adopts a merging-based strategy, our approach achieves an additional performance gain of 8.69%. This substantial improvement highlights the benefit of incorporating global performance of the merged model into the training process, rather than relying solely on post hoc model merging. Moreover, our method attains a final accuracy of 62.34%, outperforming the suboptimal baseline NSDCIL by 3.16%, consistently validating its effectiveness.

We also compare with some traditional CIL methods in Table 2, including iCaRL[Rebuffi et al., 2017], DER [Yan et al.,

2021], FOSTER [Wang et al., 2022a] and MEMO [Zhou et al., 2023]. To ensure a fair comparison , their backbones are also initialized with the pretrained ViT-B/16 model. As these methods require extra memory during incremental learning, we apply a fixed memory with 20 samples, following the iCaRL [Rebuffi et al., 2017]. Notably, our method does not require any samples during either training or inference. Experimental results show that, on the CIFAR-100 dataset, our method achieves clear performance advantages over replaybased approaches. On the more challenging ImageNet-R dataset, our method further outperforms traditional replaybased CIL methods. Overall, even without sample storage, our method demonstrates strong competitiveness against traditional CIL approaches.

5.3 Ablation Study

As shown in Table 3, we conduct ablation studies on the ImageNet-A dataset to systematically evaluate the contribution of each component relative to the baseline. We adopt ACMap as the baseline method, which trains each task independently using the cross-entropy loss and merges model parameters via an average strategy. In exp-II, we introduce the feature-level loss Lf , resulting in a performance improvement of 7.97%. In exp-III, we further incorporate the prototype loss Lproto, which achieves an additional gain of 0.39%. These results indicate that the global loss plays a critical role in enhancing the performance of the merged model. In particular, constraints imposed at the feature level improve compatibility between new and previously learned tasks, thereby effectively alleviating catastrophic forgetting. Finally, we replace the averaging strategy in ACMap with the Eq.(18), which results in a modest performance improvement. Overall, our method improves by approximately 8.69% over ACMap, further validating its effectiveness.

5.4 Long Tasks

In continual learning, models must continuously adapt to an evolving class distribution. Typically, the learning difficulty increases as the task sequence becomes longer. To evaluate the effectiveness and scalability of our proposed method under long task sequences, we conduct more challenging experiments, as reported in Table 4. In these experiments, each task in every dataset contains only two categories. For the CUB, ImageNet-A, and ImageNet-R datasets, this setting results in 100 consecutive tasks, representing a significantly more difficult learning scenario. We compare our method with ACMap, a representative merging-based approach, as well as NSDCIL, a suboptimal method that does not employ model merging. Compared with ACMap, our approach achieves superior performance across all datasets. Furthermore, relative to NSDCIL, our method achieves more pronounced improvements on the more challenging ImageNet-R and ImageNet-A datasets, with final accuracy gains of 18.12% and 15.47%, respectively. These results indicate that model merging is particularly advantageous in long task sequences and that the proposed merging-oriented bi-level optimization further enhances overall merged model performance.

We also evaluate the stability of our method as the task sequence length increases. Table 5 presents the performance of our method, compared with the baseline EASE, MOS, ACMap, and NSDCIL, across different task lengths. We conducted experiments with 5, 10, 25, and 50 tasks on the VTAB dataset. The results show that our method consistently outperforms the others on the final accuracy across all task sequence lengths, with its advantages becoming more pronounced as the task length increases. This trend indicates that our method remains effective as the number of tasks increases, demonstrating favorable scalability under long task sequences. In addition, our method maintains the highest performance stability throughout the task sequence, with the smallest overall degradation. In the B0 Inc1 scenario, our method attains a final accuracy of 92.24%, whereas the competing methods achieve a maximum of 90.03%. These results further underscore the effectiveness of our method in long-task scenarios.

6 Conclusion

In this paper, we proposed a Merging-Oriented Bi-level Optimization framework for class incremental learning (MOBO). First, we designed a global loss that treats the overall performance of the merged model as the primary training objective. Second, we adopted a bi-level optimization framework that alternately optimizes the new task parameters and the merged model parameters. The upper level updates the new task parameters using the global loss, guiding them toward a solution space that is favorable for subsequent merging. The lower level updates the merged model parameters by solving for the optimal merging coefficients. This approach dynamically balances new and old knowledge from a global perspective, effectively mitigating catastrophic forgetting and improving the final performance of the merged model. Experimental results demonstrate that our method achieves stateof-the-art performance across all datasets and yields significant improvements on long task sequences. However, while we achieved progress in solving for merge coefficients compared to prior methods, the gains remain limited. In future work, we will focus on further optimizing the selection of merge coefficients to enhance model performance.

Acknowledgments

This work was supported in part by the Fundamental Research Funds for the Central Universities under Grant 2022JBQY007, in part by the National Natural Science Foundation of China under Grant 62276019, 62306028, in part by the Beijing Natural Science Foundation under Grant L231019.