Revisiting Neural Networks for Continual Learning: An Architectural Perspective

1 Introduction

Artificial Intelligence (AI) is expected to feature a robust capability to continuously acquire and update knowledge like creatures. To achieve this, Continual Learning (CL), also known as Incremental Learning (IL), has garnered much attention in AI [Wang et al., 2023]. Early, some efforts [Li and Hoiem, 2017; Serra et al., 2018] centered on Task Incremental Learning (Task IL), where the task identity is given during both training and inference phases to assign a classifier. Recently, more works [Feng et al., 2022; Chen and Chang, 2023; Bian et al., 2024] focused on the more intricate Class Incremental Learning (Class IL), in which the task identity is accessible only during training.

∗Corresponding author.

In both Task IL and Class IL, it is essential for the model to possess the plasticity required to acquire new knowledge and the stability necessary to retain previously learned knowledge. However, the stability-plasticity dilemma remains a persistent challenge in CL [Grossberg, 2013]. To address this, existing works [Kirkpatrick et al., 2017; Rebuffi et al., 2017; Shibata et al., 2021; Wang et al., 2022] focus on developing novel algorithms to trade-off stability and plasticity. Among them, expansion-based methods dedicate different incremental network architectures to minimize conflicts between stability and plasticity. For instance, DER [Yan et al., 2021] centers on feature extractor scaling, by contrast, MEMO [Zhou et al., 2022] puts effort into specific layers. In essence, these efforts resort to network width to build a good CL model. Moreover, some works can be formulated as utilizing network components to resist forgetting, e.g., CN [Pham et al., 2022], FPF [Zhao et al., 2023], and LoRA [Lin et al., 2021]. However, these methods are built upon existing network architectures, which limits the scope of architecture exploration for better stability and plasticity.

A clear trend is that expanded architectures outperform the original ones. This raises a concern about whether the existing basic architectures of the network are ideal for CL. As shown in Figure 1, we observe that (i) various architectural designs significantly shape the impact towards CL, and (ii) directly using the native ResNet [He et al., 2016] leads to performance far from satisfactory. This motivates us to design architectures tailored for CL.

How to seek an efficient and CL-friendly architecture? Naturally, Neural Architecture Search (NAS) [Elsken et al., 2019] is a straightforward candidate for exploring architectures for CL due to the automated search that can be performed on architectures. In this paper, we employ NAS as a promising solution for CL and probe catastrophic forgetting from an architectural perspective. However, the performance of NAS hinges on the quality of the search space design [Radosavovic et al., 2020]. Given the limited prior knowledge regarding the influence of architectural designs on the performance of CL, constructing a suitable search space remains challenging. Thus, it becomes imperative to empirically glean this prior knowledge before employing NAS.

In short, this paper aims to bridge the gap between network architectures and CL. To this end, we (i) systematically scrutinize the influence of architectural design on CL, (ii) identify critical design candidates that can boost CL, and (iii) propose a NAS method with a novel search space tailored for CL.

Given that ResNet stands as a cornerstone in the AI era, we initiate this work from it to revisit neural networks and assess the impact of architectural designs on CL. Consequently, we formulated this study around network scaling and network components. Concerning the former, we consider the network width and depth. Regarding the latter, we delve into the investigation of skip connections, global pooling layers, and downsampling approaches. Subsequently, grounded in empirical observations, we craft a specialized search space for CL, each candidate architecture within this space is succinctly represented, facilitating an efficient search process. In conclusion, we propose a simple yet effective method to Craft CLfriendly Architectures, dubbed ArchCraft (AC). ArchCraft recrafts AlexNet/ResNet into AlexAC/ResAC to guide a welldesigned network architecture for CL. As a result, ArchCraft achieves superior CL performance while employing significantly fewer parameters than the original architecture.

To sum up, the contributions of this work are as follows:

• We thoroughly scrutinize the mechanisms through which neural architectural design affects CL, and demonstrate that wider and shallower architectures better align with an effective CL model.
• To the best of our knowledge, this paper is the pioneering effort to employ neural architecture design to shape a CL-friendly architecture. To achieve this, we propose a novel ArchCraft method to steer an architecture with greater stability and plasticity.
• Extensive experiments show that the proposed method is parameter-efficient across various CL scenarios, i.e., achieving state-of-the-art performance with much fewer

parameters than the baseline. Furthermore, our method features a controllable parameter.

2 Related work

2.1 Continual Learning and Neural Architectures

Continual Learning. Neural networks have achieved remarkable success in various fields [Yuan et al., 2023; Zhu et al., 2023], but are ill-equipped for CL due to catastrophic forgetting. A series of works, such as encompassing regularization, dedicated memory systems, and modular architectures, are proposed to overcome forgetting. In detail, Regularization-based methods regularize the variation of network parameters or logit outputs to maintain stability, e.g., EWC [Kirkpatrick et al., 2017] and XK-FAC [Lee et al., 2020]. Replay-based methods preserve a few exemplars from previous tasks and replay them in learning a new task to prevent forgetting, e.g., iCaRL [Rebuffi et al., 2017] and Gcr [Tiwari et al., 2022]. This category of work designs different sampling strategies to establish the limited memory buffer for replay. Expansion-based methods continually expand the network methods, in which different incremental parts of the networks are dedicated to specific tasks, e.g., BNS [Qin et al., 2021] and MEMO [Zhou et al., 2022]. This category of work focuses on the efficient utilization of network architectures.

Neural Architectures for CL. In principle, the performance of neural networks is mainly decided by their weights and architectures. The CL methods mentioned above are mainly for the weights, while the research on neural architectures for enhancing CL is still in the infant. In the literature, the work [Mirzadeh et al., 2022a] concluded that wider network architectures suffer less catastrophic forgetting in Task IL. Upon this, another work [Mirzadeh et al., 2022b] further investigated the impact of certain network components on Task IL. To the best of our knowledge, there is no systematic work dedicated to designing the CL-friendly basic architectures (i.e., the backbones).

To sum up, unlike these methods that focus on expanding or modular the specific architecture to improve CL, this work centers on the architecture itself. Notably, we extend the scope of architectural design to encompass a broader scenario, a.k.a more intricate Class IL. And we further propose an effective method for designing enhanced network architectures tailored for CL.

2.2 Neural Architecture Search (NAS)

NAS automates the design of high-performance neural architectures by formulating the design process as an optimization problem [Elsken et al., 2019]. In this process, NAS employs an optimization method (i.e., search strategy), to traverse a predefined search space comprising candidate architectures. After searching, the architecture demonstrating the best performance is chosen as the final design. Some works have already empirically shown that NAS can craft architectures that surpass those manually designed [Zoph et al., 2018].

Discussion. NAS can be employed to design architectures for CL. However, the crux of NAS lies in defining a suitable search space [Radosavovic et al., 2020; Wan et al., 2022], which delineates a searchable subset of potential architectures from the vast architecture space. While numerous architectural design practices offer substantial prior knowledge about architectures with remarkable plasticity for standard learning paradigms [Howard et al., 2019; Tan and Le, 2021], designing a search space specifically tailored for CL requires revisiting existing architectural design experiences. This is essential as architectures for CL need to possess not only plasticity but also stability [Grossberg, 2013], necessitating the construction of a CL-friendly search space.

3 Preliminaries

In this section, we describe the experimental setup for training and evaluating the networks toward CL.

Benchmark. For the CL scenarios mentioned above, i.e., Task IL and Class IL, we assess network performance on CIFAR-100. The training process involves gradually introducing all 100 classes with 5 classes per incremental step, totaling 20 steps or tasks. In Task IL, the network is trained using a vanilla SGD optimizer, while in Class IL, a replay buffer containing 2,000 examples is employed.

EvaluationMetrics. Let K be the number of tasks, the average classification accuracy after learning the b-th task, say Ab, is defined as:

where ai,b is the classification accuracy evaluated on the test set of the i-th task after learning the b-th task (i ≤ b). The performance of CL is measured by two metrics: the Last Accuracy (LA) and the Average Incremental Accuracy (AIA). The LA is the classification accuracy after the last task, i.e., LA = AK, which reflects the overall accuracy among all classes. Further, the AIA denotes the average of Ab over all tasks, i.e., AIA = 1K ∑︁Kb=1 Ab, which reflects the performance of all incremental stages. The higher LA and AIA, the better CL performance.

Implementation Details. In all experiments, we report the mean and standard deviation calculated across 5 runs with different random seeds. For Task IL, we train the model by 60 epochs in the first task and 20 epochs in the subsequent tasks. For Class IL, we follow PyCIL [Zhou et al., 2023] to train the model by 200 epochs in the first task and 70 epochs in the subsequent tasks.

4 Design of CL-Friendly Architectures

In this section, we first decompose and study the architectural design on the performance of Task IL and Class IL at the network components and scaling. Upon these insights, we propose an ArchCraft method to craft CL-friendly architectures.

4.1 Impact of Network Components

Strided convolution, skip connections, and Global Average Pooling (GAP) have all proven effective in neural networks in standard learning paradigms [He et al., 2016]. However, the impact of these components on CL remains elusive. To address this gap, we perform experiments to determine the optimal configurations of these components for both Task IL and Class IL. To be specific, the candidate configurations can be obtained by replacing strided convolution with average/max pooling, removing skip connections, or removing GAP. In particular, we utilize the ResNet series as the architectural framework and examine the impact of the aforementioned components on CL performance.

As shown in Table 1, different configurations of network components influence the performance of both Task IL and Class IL. In brief, we observe that the optimal configuration involves employing max pooling and skip connections while removing GAP for Task IL. Moreover, the optimal configuration for Class IL is similar to Task IL but retains GAP. In particular, skip connections significantly improve the performance in both Task IL and Class IL, mirroring their role in standard learning paradigms. In contrast, the function of GAP diverges significantly between Task IL and Class IL. In Task IL, eliminating the GAP in networks considerably enhances CL performance, while in Class IL, it causes severe performance degradation.

4.2 Impact of Network Scaling

Width and depth are two crucial factors that affect the capacity and complexity of neural networks. We conduct a series of experiments on networks with different widths and depths to investigate the impact on CL performance. Specifically, we adopt ResNet-32 as the skeleton of the network [He et al., 2016], which consists of a single convolution layer and three subsequent stages, each of them containing the same number of blocks. In that case, the width is represented by the initial number of channels, denoted by W, and the depth is represented by the number of blocks in each stage, denoted by D. Moreover, we note that all of the network architectures are modified by applying the optimal configurations of network components summarized in Section 4.1.

Figure 2 compares the CL performance of networks with different W and D. We observe that in both Task IL (see Figure 2a and 2b) and Class IL (see Figure 2c and 2d), the network with a larger width tends to exhibit better performance of CL in most cases. This phenomenon cannot be simply attributed to over-parameterization, as increasing depth does not yield similar results. In contrast, increasing the depth of the networks can even lead to obvious performance degradation in Task IL (see Figure 2a and 2b). These results indicate that wider and shallower networks may be more suitable for CL, explaining why ResNet-18 empirically shows better performance than ResNet-32 in Table 1. Therefore, we conclude that scaling the network depth and width appropriately is a potential way to enhance CL performance.

The above results raise a concern about whether the increase in the overall network width (i.e., initial width) or the width of the classifier (i.e., final width) contributes to the improvement of CL performance. To investigate this, we conduct further experiments on networks with the same final width but varying initial widths. In detail, we consider the network architectures with D = 1. To allow for adjustments to the final width, an additional convolutional layer with a 1 × 1 kernel size is inserted before the classifier for these networks. We maintain the final width of the network with W = 64 constant and adjust the output channels of the inserted convolutional layer for other networks to match it. The CL performance of networks with different initial and final widths are reported in Table 2.

As shown in Table 2, the results of "performance in Task IL" indicate that increasing the classifier width significantly benefits Task IL. Moreover, networks with different initial widths but the same final width show comparable performance of CL. This implies that the classifier width, rather than the overall network width, is the main factor affecting the performance of wider networks in Task IL. It also suggests that the adverse effect of GAP on Task IL may stem from the reduction in the width of the classifier. By contrast, we observe from the results of "performance in Class IL" that the impact of initial width on Class IL is as essential as classifier width. This indicates that the overall network width and the width of the classifier both significantly affect Class IL.

4.3 ArchCraft Method

To further investigate architectural designs that benefit CL, we propose ArchCraft, a straightforward yet powerful NAS method for CL. ArchCraft comprises a CL-friendly search space and employs a simple search strategy grounded in genetic algorithms.

Search Space towards ArchCraft

What architectural design elements can be predetermined based on the aforementioned findings? The above experiments show that ResNet can achieve good CL performance with slight modification, which demonstrates the great generalizability of its overall architecture for CL. Therefore, in our designed CL-friendly search space, the overall architecture of the networks is predetermined following the design of ResNet, i.e., each network architecture consists of a stem (i.e., a convolutional layer) and several units. Since the promising architecture is relatively shallow according to the experiments in Section 4.2, we use a simple unit that comprises a single convolutional layer and a skip connection to explore more flexible architecture. More specifically, each convolutional layer is equipped with a 3 × 3 convolutional kernel and followed by batch normalization and ReLU. In addition, the network components are determined according to the corresponding optimal configurations summarized in Section 4.1 for Task IL and Class IL, respectively.

What architectural design elements require further exploration? To achieve a better CL performance, further exploration of certain architectural design elements is still necessary. First, although the significance of network width and depth for CL has been demonstrated, determining their optimal values necessitates further investigation. Second, the positions at which the channel number is increased and feature map down-sampling occurs are critical for CL performance, as they significantly influence the feature extraction capability of the network. However, their impact on CL has not been thoroughly examined in prior experiments. Therefore, the optimal designs of the locations of these operations are still necessary to search for. In summary, the architectural variations within our proposed search space encompass network width, depth, locations for increasing the channel number, and locations for down-sampling the feature map.

How to concisely represent the candidate network architectures in search space? We suggest an effective scheme to encode the architectures into a concise form that facilitates searching and adaptive scaling of parameters. The number of units and the initial number of channels are each denoted by a single code, which respectively represents the network depth and width. We specify that the size of the feature map can be halved zero to five times at chosen locations, which is indicated by five codes. Specifically, if the code x is less than the number of units, it indicates that the size of the feature map is reduced by half through max pooling before the (x + 1)-th unit; otherwise, it holds no significance. Similarly, the locations of the units in which the number of output channels is doubled are indicated by the other five codes. By the above means, each architecture in the proposed search space can be simply yet concisely encoded as a sequence comprising 12 codes. Additionally, the parameters of networks can be easily scaled by simultaneously reducing the network width and depth when the number of parameters exceeds the limit.

Search Strategy towards ArchCraft

In our evolutionary search strategy, the search process consists of an initialization phase and iterative evolutionary phases. During the initialization phase, we randomly generate a population of candidate architectures (i.e., individuals) within the defined search space. Subsequently, the fitness is evaluated for each individual. In each iteration of the evolutionary phase, the offspring is generated through the mutation after fitness evaluation. In detail, to generate an offspring individual, (i) two individuals are randomly selected from the population, and the one with higher fitness is chosen as the parent, (ii) the parent is copied and a randomly selected code of it is modified. In particular, when the code representing the number of units is modified, the codes that indicate the locations to down-sample feature maps and raise channel numbers are adjusted accordingly to keep the relative positions fixed. After the offspring generation, individuals are evaluated for fitness, and those with higher fitness are selected from the previous population and offspring to form the next population. Upon completion of the search process, the architecture exhibiting the highest fitness is chosen as the final design.

Performance Evaluation towards ArchCraft

We employ a direct method to evaluate the performance (i.e., fitness) of searched architectures. Specifically, each architecture is trained and assessed with the settings described in Section 3, with the AIA serving as the fitness metric.

5 Evaluation of the Improved Architectures

We apply the proposed ArchCraft method to design improved architectures for Task IL and Class IL separately, with a population size of 10 and an evolution of 20 generations. Then, we empirically compare these architectures with the baseline ones to demonstrate the efficacy of ArchCraft.

5.1 Better CL Performance

Benchmark

We choose CIFAR-100 and Imagenet-100 to evaluate the ArchCraft-guided architectures. Both datasets are divided into 20 tasks of 5 classes each and 10 tasks of 10 classes each to construct four benchmarks: C100-inc5, C100-inc10, I100-inc5, and I100-inc10. It should be highlighted that since the ArchCraft-guided architectures are crafted based on C100-inc5, the experiments conducted on C100-inc10 and ImageNet-100 serve to validate their generalizability.

Evaluations in Task IL

AlexAC. AlexNet [Krizhevsky et al., 2012] is widely used as the basic architectures in Task IL [Serra et al., 2018; Konishi et al., 2023]. Thus, we craft two architectures with distinct parameter sizes based on them: AlexAC-A and AlexAC-B featuring marginally and far fewer parameters than AlexNet.

Setup. For CIFAR-100, we compare ArchCraft with several classical Task CL methods: SI [Zenke et al., 2017], HAT [Serra et al., 2018], WSN [Kang et al., 2022], and SPG [Konishi et al., 2023]. To make this comparison, we train AlexAC-A and AlexAC-B using a vanilla SGD optimizer, while AlexNet is trained with various CL methods. Additionally, we report the upper-bound performance for all networks. Note that the term 'Upper-bound' refers to the result of training a separate model for each task, thereby eliminating any forgetting. Furthermore, we also provide a comparison between AlexNet and AlexAC-A on Imagenet-100, both of which are trained using a vanilla SGD optimizer.

Table 3 details the comparison between the AlexAC series and AlexNet on CIFAR-100. It is evident that the AlexAC series consistently outperforms AlexNet in all cases. In particular, when both are trained using the vanilla SGD, AlexAC-A achieves 28.6% and 16.99% higher LA than the AlexNet on C100-inc5 and C100-inc10 settings, respectively, while utilizing fewer parameters. The magnitude of this improvement surpasses that achieved by state-of-the-art methods (28.6% and 16.99% vs. 23.12% and 12.38%). These results underscore the pivotal role of network architectures in CL. This implies that advancements in CL can be propelled not only through learning methods but also through improved architectural designs. Moreover, AlexAC-B attains superior performance compared to state-of-the-art methods with 86% fewer parameters. In conclusion, these observations strongly emphasize the significance of enhanced architecture in CL.

Table 4 reports the comparison between AlexAC-A and AlexNet on Imagenet-100. We observe that AlexAC-A also achieves significantly higher LA (16.66% and 12.48%) with fewer parameters compared to AlexNet. This result demonstrates that ArchCraft can design generally useful architectures that facilitate Task IL.

Evaluations in Class IL

ResAC. ResNet series are widely used as the basic architectures in Class IL. Thus, we craft two architectures with distinct parameter sizes based on them: ResAC-A and ResAC-B featuring fewer parameters than ResNet-18 and ResNet-32.

Setup. We evaluate our method combined with several classical Class CL methods: Replay, iCaRL [Rebuffi et al., 2017], WA [Zhao et al., 2020], and Foster [Wang et al., 2022]. Hyper-parameters for all methods adhere to the settings in the open-source library [Zhou et al., 2023], and a fixed memory size of 2,000 exemplars is utilized for all methods.

Table 5 reports the LA and AIA of the ResAC series and ResNet series in Class IL. In this case, ResAC-A consistently outperforms ResNet-18 across all methods, datasets, and incremental steps. In particular, ResAC-A achieves a maximum 8.19% higher LA and 8.02% higher AIA with 23% fewer parameters than the ResNet-18. Moreover, ResAC-A demonstrates superior performance on ImageNet, indicating robust generalizability. This suggests that the enhancement afforded by the improved network architecture can effectively generalize to various methods and settings. Furthermore, under stricter parameter constraints, ResAC-B, compared to ResNet-32, attains up to 5.72% and 4.98% improvements in LA and AIA with 4% fewer parameters. In conclusion, the proposed method can serve as a valuable complement to CL.

5.2 Reducing the Number of Parameters

In this subsection, we thoroughly examine the effects of parameter sizes and network architecture. To simplify, we choose one typical method Foster as a baseline.

First, Figure 3a yields three conclusions: (i) ArchCraft can craft neural networks with controllable parameter sizes, e.g., from ResAC-0.18M to ResAC-8.63M. (ii) ResAC series achieve better CL performance with significantly fewer parameters than the naive ResNet series, e.g., with similar performance, 61% and 97% fewer parameter sizes (from 0.46M

to 0.18M and from 11.17M to 0.33M, respectively); With similar parameter sizes, 2.3% higher performance on AIA (from 67.14% to 69.44%), or more. (iii) Existing state-of-theart NAS methods, e.g., DARTS+PT [Wang et al., 2021] and PA&DA [Lu et al., 2023], fail to design CL-friendly architectures. For this point, we think that most existing NAS methods can not adaptively adjust the parameter sizes, since the key architectural elements (i.e., depth, width) that determine the number of parameters rely on predefined manually in their search space [Liu et al., 2018; Wan et al., 2022]. In summary, ArchCraft recrafts AlexNet/ResNet into AlexAC/ResAC to guide a well-designed network architecture for CL with fewer parameter sizes.

In addition, as Figure 3b, we compare the proposed method with the expansion-based method MEMO [Zhou et al., 2022]. As is known to all, this category usually obtains performance improvement at the expense of increasing the parameter size of the baseline (from 11.17M to 86.72M). This leads to the limited availability of methods of this category. In radical contrast, ArchCraft achieves a better result with much fewer parameters than the baseline architecture (from 11.17M to 8.63M). To sum up, these empirical results provide fresh insights for advancing CL.

5.3 Stronger Stability and Plasticity

To further scrutinize the effectiveness of ArchCraft, we report the average forgetting and accuracy of the new task on C100-inc10 for Task IL and Class IL. As illustrated in Figure 4, the ArchCraft-guided architectures exhibit less forgetting on the previous task and higher accuracy on the current task than the baselines in a consistent setting. This observation suggests that stability and plasticity can be concurrently improved by employing architectures designed by ArchCraft. Furthermore, it emphasizes the critical role of heightened stability in enhancing CL performance.

What contributes to the stronger stability of the ArchCraft-guided network architecture? To investigate this, we quantitatively assess the similarity of network representations across models from all incremental stages during CL. In detail, we utilize Centered Kernel Alignment (CKA) [Kornblith et al., 2019], a robust and widely used metric, to compute the similarity between representations. To facilitate this comparison, we extract feature maps from the final convolutional layer of each model by inputting the same set of instances. Subsequently, CKA is applied to measure the similarity between these feature maps.

To put it more intuitively, we present the similarity matrix in Figure 5. Figure 5 illustrates that features extracted by the ArchCraft-guided network architecture show a higher degree of similarity across all incremental stages compared to the baseline ones. This observation suggests that the network architecture guided by ArchCraft more effectively extracts shared features between incremental tasks, facilitating adaptation to new tasks with minimal changes. Thus, ArchCraft results in better overall stability while maintaining plasticity.

Figure 5: CKA visualization of the ArchCraft-guided networks and the baseline ones in different incremental stages across.

6 Conclusion

This paper identifies specific architectural designs that influence CL. The proposed ArchCraft bridges the gap between network architecture design and CL, achieving superior CL performance while utilizing a significantly more compact number of parameters than naive CL architectures. We affirm that ArchCraft holds immediate practical relevance in the design of CL-friendly networks. We hope this work inspires further study on overcoming catastrophic forgetting and enhancing CL performance from the architectural perspective.

Acknowledgements

This work was supported by National Natural Science Foundation of China under Grant 62276175.