Knowledge Consolidation based Class Incremental Online Learning with Limited Data

1 Introduction

Deep neural networks have achieved promising results on various tasks. However, these models suffer from the problem of catastrophic forgetting [Kirkpatrick et al., 2017]. The most prominent reason for catastrophic forgetting is that the model is not trained to also remember the previous knowledge when acquiring new knowledge. In general, the model is trained to optimize its performance on the current task with no consideration of how the updated model will perform on earlier tasks. This greedy update overwrites the parameter values that may have been optimal for previous tasks. Continual-learning (CL)

∗Contact Author

(also sometimes referred to as lifelong/incremental learning) is a learning paradigm to address this issue in deep neural networks and has been gaining significant attention in recent work [Parisi et al., 2019].

We present a novel approach for class incremental online learning problem in a limited data setting. This problem setting is more challenging than standard class incremental learning [Javed and White, 2019] due to additional constraints: (1) Data in each class appears in the online fashion, i.e., the model sees every training example exactly once; (2) The number of training examples in each class is very small; and (3) We do not use any replay/memory to store the training examples from previous classes. This is the most general setting for class incremental learning and various practical usage scenario can be obtained through this or a relaxed setting. For instance, in face recognition, it is common to have few examples per class but usually not in an online learning fashion, whereas for a robot navigating in an environment, the setting would also be online. We empirically show that learning a robust representation that can accommodate future tasks may be a potential solution to handle the problem mentioned above. Our proposed approach achieves this by leveraging the meta-learning [Finn et al., 2017] framework with knowledge consolidation.

Meta-learning [Finn et al., 2017] has proven to be an effective approach for learning generic feature representations that can be rapidly adapted to new tasks by fine-tuning using very few examples (and in some cases, even without finetuning [Vinyals et al., 2016; Raghu et al., 2020]). While such use of meta-learning might seem appealing and does indeed show some promising results in continual learning settings [Javed andWhite, 2019], in practice, this approach is still prone to the problem of catastrophic forgetting. One of the reasons for this is the overparametrized nature of deep neural networks, in which only a few neurons are activated/fired for all samples. As a result, the network is reliant only on a small set of parameters. Although this may not be a problem when learning only a single task, it can potentially be an issue in continual learning where we are required to learn a sequence of tasks and, while learning a new task, any changes to these parameters can drastically affect the performance on the older tasks.

To address this, we present a knowledge consolidation based meta-learning approach. In our approach, during training, we identify and split the network parameters into two groups, called important and unimportant. The network's existing knowledge is squeezed into the set of important parameters, and the unimportant/dead parameters are freed, thereby expanding the network's capacity to accommodate the future learning trajectories. Briefly, a learning trajectory is a sequence of examples where examples from a particular class occur together in the sequence (we discuss this in detail later). The proposed strategy ensures that the knowledge from the old learning trajectories is preserved in a compressed form within a small set of important network parameters, which we identify and isolate, and then move on to adopt new learning trajectories. The extra knowledge obtained via learning from the new trajectories updates the unimportant parameters, and they also become important. In knowledge consolidation, we rejuvenate the dead neurons in the model and consolidate the knowledge of the previously preserved parameters and the new rejuvenated parameters. This helps to learn a robust representation. Therefore, the model capacity is fully utilized, and small changes in a few parameters strongly resist the catastrophic forgetting. The knowledge consolidation process overview is shown in Fig. 1.

Note that we use knowledge consolidation and metalearning in the training phase. Training is done on a base class set using multiple learning trajectories. We strictly follow class incremental online learning setting within a learning trajectory. However, across learning trajectories, the class incremental learning setting is not used (since multiple learning trajectories can have the same set of classes). Therefore, we do not follow the incremental learning setting during training on the base class set (since training is done on multiple learning trajectories). We follow this to make the model's representation robust and facilitate future continual learning during evaluation time (Section 2), where we perform class incremental online learning with limited data on a novel class set. We use entirely different (disjoint) classes in the novel class set than the base class set, and during evaluation on the novel class set, we only use meta-learning for quick adaptation.

Our approach significantly outperforms other incremental learning methods by a significant margin. We show that a basic online updating strategy on representations learned by our approach Knowledge Consolidation based Class Incremental Online Learning (KCCIOL) is better than memory-based rehearsal methods. Our approach can also be integrated with existing continual learning approaches such as MER, EWC, ER-Reservoir as shown in Section 7.

2 ProblemFormulation and Evaluation Protocol for Novel Class Set Testing

Let {τ1, τ2 . . . τk . . . τl . . . } denote a stream of learningtrajectories and ∀i, τi ∼ Ptest(τ) where Ptest(τ) denotes the trajectory distribution during testing from the novel class set. Each learning-trajectory τi is further split into two sets - train and validation, i.e., τi = {τtr, τval} where τtr = {xn, yn}kn=1 and τval = {xn, yn}k+s n=k+1 are the labeled samples, {k, s} ∈ N. Here, ∀n, (xn, yn) ∈ (Xtest, Ytest) denote the input and label pairs and the trajectory distribution Ptest(τ) is defined over (Xtest, Ytest) which is disjoint from base training class set (Xtrain, Ytrain) i.e. Ytrain ∩ Ytest = ∅. Moreover, we assume class(τtr) = class(τval) i.e. classes of τtr are the same as classes of τval. The goal of continual learning is to minimize the loss on the unseen examples of classes learnt earlier in an incremental fashion, and can be written as: Eτ∼Ptest(τ )[L(f (τ x val |θ, W), τ y val )] i.e model is evaluated on τval. Our evaluation protocol (Algorithm 4) is similar to the class-incremental setting [Javed and White, 2019], but samples within each class also arrive in an online manner. In particular, here are the key differences: 1) We assume availability of very few samples per class; 2) For a particular class, each sample is seen exactly once; and 3) We do not use any replay mechanism. These differences make our problem setting considerably more challenging than the standard class-incremental setting.

3 Training on Base Class Set usingMeta Learning Approach

Following a similar notation as in Sec. 2, let {τ1, τ2 . . . τk . . . τl . . . } denote a stream of learning trajectories and ∀i, τi ∼ Ptrain(τ) where Ptrain(τ) denotes the learning trajectory distribution during training. Following the model-agnostic meta learning (MAML) set-up [Finn et al., 2017], we assume that the ith learning-trajectory's data τi is further split into two sets, meta-train and metaval, i.e. τi = {τtr, τval} where τtr = {xn, yn}kn=1 and

τval = {xn, yn}k+s n=k+1 are the labeled samples, {k, s} ∈ N. Here, ∀n, (xn, yn) ∈ (Xtrain, Ytrain) denote the input and label pairs and the learning trajectory distribution Ptrain(τ) is defined over (Xtrain, Ytrain). Moreover, we assume class(τtr) ⊂ class(τval) i.e. classes of τtr are a proper subset τval; therefore τval contains all the classes of τtr and some additional classes. Note that τval and τtr are the random trajectories of length k and s, respectively. In τtr, samples of each class occur together, i.e samples from class 2 occur after and before samples from class 1 and class 3, respectively. We sample a learning trajectory multiple times for training on a base class set. We randomly sample a learning trajectory by selecting a subset (proper) of classes (randomly) from the base class set. Therefore, every learning trajectory has a different class order.

We follow the MAML setting of continual learning [Javed and White, 2019]. In the inner loop, each class arrives in a sequential manner, and the boundary of each class is available in advance. In the outer-loop, the aim is to minimize the empirical risk over the unseen data, provided the model is optimized on the seen data in an online fashion with the continual learning constraint i.e. when learning from the training data of current class, we are not allowed to access the training data of previous classes. The overall loss function across all trajectories is defined as:

L(f (τ x val |θ, W), τ y val ) denotes the loss on the model f for the validation trajectory of τ . For notational brevity (and slight abuse of notation), we use τ x val to refer to all the inputs of validation trajectory of task τ and τ y val to refer to the corresponding true labels.

The function f : X → Y is defined as f(τ x |θ,W) = g(h(τ x |θ)|W) where hθ : X → Rd is defined by parameter θ (representation learning parameters) and gW : Rd → Y is defined by W. The classifier parameters W are learned using meta-train set (τtr), and representation learning parameters θ andWare jointly learned using the meta-val set (τval).

In the inner-loop of the meta-learner, which learnsW, the model is trained on the meta-train data τtr. The outer loop, which learns θ and W is trained using meta-val data τval. In the outer loop, the model's loss is also computed on novel classes not seen during the inner-loop training since the classes in τtr are a proper subset of classes in τval. Evaluation on both sets τtr and τval makes the model perform well on both current and previously learned classes. The optimization problems solved by the inner loop and the outer loop are given by:

In Eq. 2, for notational simplicity, we use the entire training data but during training we perform an online update over τtr. The above two optimization problems are solved in an alternating fashion using τtr and τval, respectively, with the most recent parameter W obtained from Eq. 2 used in Eq. 3.

4 Knowledge Consolidation basedMeta Learning for Training on Base Class Set

While the meta-learning based approach for continual learning described in the above section is promising, it is not particularly effective for our problem setting. One of the reasons for this is the overparametrized nature of deep neural networks, in which only a few neurons are activated/fired for all samples. As a result, the network is reliant only on a small set of parameters. Although this may not be a problem for a single task learning setting, it can potentially be an issue in continual learning where we are required to learn a sequence of tasks and, while learning a new task, any changes to these parameters can drastically affect the performance on the older tasks. We overcome this by using a knowledge consolidation based meta-learning approach. Our proposed approach identifies the important and unimportant/dead parameters, rejuvenates the dead parameters, and consolidates the knowledge of the important and reborn parameters (Fig. 1). Therefore, the model capacity is fully utilized, and small changes in a few parameters strongly resist the catastrophic forgetting.

A simple way to assess the importance of a parameter is to use its absolute value [Han et al., 2015], often used in deep model compression. We can discard weights/parameters having small absolute values without sacrificing upon the model's performance [Han et al., 2015]. We leverage this simple idea to identify the important parameters in the model effectively. The proposed approached modifies the meta-learning framework by introducing knowledge consolidation. We define A = [θ,W] as the joint set of parameters of the complete model. We partition model parameters A into two disjoint sets. The important parameters are denoted by AI and the "less important" ones are denoted by AL, s.t., A = {AI , AL} and AI ∩ AL = ∅.

Most of the model's knowledge is contained in AI. Our goal is to preserve the knowledge present in AI . We apply a weight-constrained regularization on AI to ensure minimal changes when a new trajectory is learned. On the other hand, we let AL be free to change in order to accommodate new trajectories. Therefore, while learning a new set of trajectories, the following regularized loss function is optimized:

One way to define the weight-constrained regularization R(AI) would be λ||At+1 I -AtI ||F , whereAtI is the important weights after tth step. The large value of λ ensures minimal changes in the important weights.

Na¨ıvely partitioning the model into {AI , AL} (based on absolute value) often does not show any significant improvement since various techniques like dropout and batch normalization force the model's knowledge to be shared across all model parameters, which causes AL to contain non-negligible knowledge. Ideally, the value of unimportant weights should be zero. However, in reality, this is not the case and the set AL usually contains the non-negligible information. Therefore, we first distill the model's knowledge into a subset of parameters A∗I (important parameters set) such that the remaining part A∗U (unimportant parameters set) contains negligible information.

The weight-constrained regularization can now be imposed on A∗I , while the set A∗U is free to be adapted for the new trajectories. To transfer/distill the model's knowledge to a subset of the parameters, we finetune the complete model with the following ℓ1 regularized objective:

We can maintain model performance by using an appropriate hyperparameter γ. The ℓ1 regularizer forces the model knowledge to be squeezed in a subset of model parameters A∗I . Rest of the model parameters A∗U contain negligible information and therefore are free to change. Now, the set A can be split into important parameters A∗I and unimportant parameters set A∗U , i.e., A = {A∗I , A∗U } and A∗I ∩ A∗U = ∅. Given this updated set of important and unimportant/dead set of parameters, the outer-loop optimization of the meta-learner is given by (akin to Eq-4)

To preserve the knowledge contained in A∗I , we apply weightconstrained regularization on A∗I as above, which ensures that A∗I do not change drastically when new learning-trajectories are encountered. Rest of the parameters (A∗U ) are free to change. Therefore, we "rejuvenate" the parameters in A∗U . Representing these rejuvenated set of parameters as A∗R, now the consolidated knowledge from both A∗R and A∗I provides a robust representation for our problem setting. Therefore, small changes in a few parameters strongly resist catastrophic forgetting because the model capacity is fully utilized, and model predictions are not reliant on a small set of parameters. As demonstrated by our experiments, such a parameter rejuvenation and knowledge consolidation significantly enhance the performance of a meta-learner based model for class incremental online learning. For a summarized algorithmic description of our approach, please refer to the Algorithms 1, 2, 3, 4.

5 RelatedWork

Incremental Learning Methods: Rehearsal based incremental learning methods [Rebuffi et al., 2017; Shin et al., 2017; Chaudhry et al., 2019; Isele and Cosgun, 2018; Rolnick et al., 2019; Chaudhry et al., 2021] store a part of training data and re-train on this stored data while training on new tasks. Regularization based incremental learning methods [Lopez-Paz and Ranzato, 2017; Kirkpatrick et al., 2017; Chaudhry et al., 2019] add an additional regularization term in the loss function, which prevents the weights from changing drastically when moving from one task to the next task. DER [Buzzega et al., 2020] uses both rehearsal and regularization. PODNet [Douillard et al., 2020] uses a spatial distillation-loss along with a representation made of proxy vectors from each class. The dynamic network methods [Singh et al., 2020; Singh et al., 2021] are also proposed for incremental learning.

Algorithm 1 Training Algorithm

Require: γ: Learning coefficient for L1 loss

Require: λ: Learning coefficient for constraint loss

Require: δ: Fraction of model parameters ( |A∗I | |A| )

Require: {αi, βi}3i=1 : Inner and outer loop lr (learning rate) Require: {stepsi}3i=1 : No of steps for KCCIOL algorithm

• 1: Randomly Initialize model parameters Θ,
• 2: mask = 0
• 3: Θ =KCCIOL(α1, β1, 0, 0,mask,Θ, steps1)
• 4: Θ =KCCIOL(α2, β2, 0, γ,mask,Θ, steps2)
• 5: mask = GetMask(Θ, δ)
• 6: Θ =KCCIOL(α3, β3, λ, 0,mask,Θ, steps3)

Algorithm 2 KCCIOL

Algorithm 3 Mask Calculation

Require: Θ: Model Parameters

Require: δ: Fraction of model parameters ( |A∗I | |A| )

• 1: procedure GETMASK(Θ, δ)

2: threshold = percentile(|Θ|, 1 - δ) 3: mask = zeros(len(Θ)) 4: index = 0 5: while index <= len(Θ) do 6: if mask[index] >= threshold then 7: mask[index] = 1 8: index + + 9: return mask

Algorithm 4 Evaluation Protocol

Require : ptest(τ): Distribution over learning trajectories Require : θ: Representation Learning Parameters

• 1: Randomly Initialize W
• 2: Sample learning trajectory τi ∼ ptest(τ)
• 3: {τtr, τval} = τi
• 4: for j =1, . . , k do
• 5: (Xj , Yj ) = (τ x tr [j], τ y tr [j])
• 6: Wj =Wj-1 -α∇Wj-1 [L(f(Xj|θ,Wj-1), Yj)]
• 7: return Accuracy(f(τ x val |θ, Wk), τ y val )

Meta-Learning Methods: MER [Riemer et al., 2019] uses meta-learning to learn parameters that prevent interference and enable knowledge transfer based on future gradients. OML [Javed and White, 2019] is a meta-learning approach that focuses on learning a generic representation that prevents catastrophic forgetting during online learning. OSAKA [Caccia et al., 2020] is a general approach to continual learning where the agent must be able to solve previously unseen distribution tasks with minimal forgetting on previous tasks. OSAKA is different from the standard incremental learning setting as seen tasks can be revisited, and online average accuracy is reported at the end of the training instead of reporting accuracy on all seen tasks. Therefore, it would be unfair to compare our method with OSAKA.

Online Continual Learning Methods: MERLIN [K J and Balasubramanian, 2020] is a replay based method for online continual learning which learns a meta-distribution from which task specific parameters are sampled at the time of inference. [von Oswald et al., 2020] proposed a method similar to MERLIN where deterministic task specific weights are generated using hypernetworks [Ha et al., 2017]. GSS uses constrained optimization for sampling samples for replay instead of random sampling for online continual learning. Incremental learning in online scenario [He et al., 2020] tackles the problem of catastrophic forgetting in the online scenario under different setting than ours. [He et al., 2020] is a memory based method where future data consists of samples from new classes as well as unseen samples from old classes, whereas, in our approach, data from one class are seen together. CTN [Pham et al., 2021] is a bi-level optimization network that uses a context network to model task-specific features which address catastrophic forgetting and knowledge transfer. But the context network needs task-specific knowledge and semantic memory to function, whereas our method does use replay and is task-free.

6 Experiments

We evaluate our approach via extensive experiments across various datasets. We follow the evaluation protocol where the model is updated in an online fashion and later evaluated on the unseen data (Section 2). We compare the performance of our model (KCCIOL) against several baselines.

Baselines: In the Scratch baseline, we evaluate the performance of a randomly initialized network. In Pretrained baseline, the network is pretrained on the train set. The SRNN approach uses a Set-KL method proposed by [Liu et al., 2019] to learn a sparse representation using train set. MRCL is a recent approach originally proposed by [Javed and White, 2019] to train a model in a meta-learning setup for continual learning tasks. OML is a modified version of MRCL proposed by [Javed and White, 2019] where the classifier parameters are randomly re-initialized at each step of the training. MAMLRep is also a MAML [Finn et al., 2017] based algorithm similar to OML and MRCL, where batch updates are performed in the inner-loop.

6.1 Experiments on Omniglot Dataset:

Implementation Details: The Omniglot dataset [Lake et al., 2015] contains 1623 classes of different handwritten characters from 50 different alphabets. Each class contains 20 samples with 15/5 as the train/test split. The first 963 classes constitute the (Xtrain, Ytrain) and the remaining classes are used as (Xtest, Ytest). For learning trajectory during training, τtr consists of 10 samples from a class randomly sampled from the training set. τval consists of 10+1 samples where ten samples are randomly sampled from the train set, and the 11th sample belongs to the class used in τtr. During the evaluation, each learning trajectory consists of an equal number of classes in both τtr and τval which are sampled from the test set. We use 15/5 samples per class for τtr/τval during evaluation. We use Adam optimizer, and the first six layers are used for learning representation.

Hyperparameter Settings: We train our model using hyperparameters: β1 = 1e-4, α1 = 1e-2, steps1 = 20000, β2 = 1e-4, α2 = 1e-2 , γ =5e-5, steps2 = 15000, β3 = 1e-4, α3 = 1e-2 , λ =5e-4, steps3 = 4000, δ = 0.5.

Model Architecture: We use six convolutional layers followed by two fully connected layers, and each convolutional layer contains 256 filters of 3 × 3 kernel size with (2, 1, 2, 1, 2, 2) strides (same as used in [Javed and White, 2019]). ReLU activation function is used for the non-linearity.

The proposed approach KCCIOL performs better by a significant margin on a wide range of classes as compared to the other baselines. We achieve a significant performance boost of 13.3 %, when 200 classes are learnt continually as shown in Table 1.

6.2 Experiments onMini-Imagenet Dataset

Implementation Details: Vinyals et al. [Vinyals et al., 2016] proposed the mini-imagenet dataset, which is a subset of the imagenet dataset. There are a total of 100 classes with 600 colored images of size 84 × 84. We use 64 classes for training and 20 classes for testing. For learning trajectory during training, τtr consists of 10 samples from a class randomly sampled from the training set. τval consists of 15 samples where 10 samples are randomly sampled from the training set, and 5 samples belong to the class used in τtr. During the evaluation, we sample classes from the test set for creating a learning trajectory. We use 30 samples per class for τtr and τval. We use Adam optimizer, and the first six layers are used for learning representations.

Hyperparameter Settings: We train our model using hyperparameters: β1 = 1e-4, α1 = 1e-1, steps1 = 26000, β2 = 1e-4, α2 = 1e-1 , γ =1e-4, steps2 = 26000, β3 = 1e-4, α3 = 0.5 , λ =1e-4, steps3 = 22000, δ = 0.5.

Model Architecture: The model architecture for the MiniImagenet is same as used in the Omniglot dataset.

Our method KCCIOL consistently outperforms others as shown in Table 2. We get a performance boost upto 12.4 % over the OML.

7 KCCIOL Complements Existing Continual Learning Methods

We have demonstrated the efficiency of KCCIOL when learning-trajectories are learnt in a continual online fashion. In this section, we will demonstrate that the representation generated by KCCIOL not only performs well on online updates but it also greatly improves the performance of other continual learning methods such as MER [Rolnick et al., 2019], EWC [Kirkpatrick et al., 2017], ER-Reservoir [Rolnick et al., 2019] when they use our model as base network. Online, Approx IID baselines have been taken from [Javed and White, 2019].

We evaluate the model's performance under four different settings for each continual learning method, which are Standard, Pre-Training, OML, and KCCIOL. In OML and KCCIOL, we use the θ generated by these methods as the base of the model and do not update the θ parameters. In the Pre-Training setting, we train the model independently on the training data and again keep the θ fixed during the process. The difference between the Pre-Training and Standard setting is that all the parameters of the model are updated in the case of the Standard setting, while in the pre-training, only W are updated. In the standard setting, the model is pretrained independently on train data to prevent other settings from having an unfair advantage of being trained on the training set. From Tables-3, 4 we can observe that KCCIOL with just online updates significantly outperforms other continual learning methods even when they are combined with OML. KCCIOL can do so because it can learn a generic robust feature representation that is generalizable across all tasks, enabling the model to perform future learning. For the train/test split, we use 15/5 samples per class. Even without using any memory/replay, KCCIOL can outperform the replay based methods by a significant margin.

8 Performance Comparison with Recent SOTA Incremental Learning Methods

We also compare our method KCCIOL with the recent stateof-the-art incremental learning methods on the mini-imagenet dataset under incremental online learning setting. From Table 5, we can observe that recent incremental learning methods such as DER [Buzzega et al., 2020], PODNet [Douillard et al., 2020], HAL [Chaudhry et al., 2021] perform poorly in an incremental online setting. Even SOTA methods for online learning, such as GSS [Aljundi et al., 2019], MERLIN [K J and Balasubramanian, 2020] suffer from catastrophic forgetting when training data is limited. We only use thirty samples per class for experiments on the mini-imagenet dataset. We observe that KCCIOL outperforms the next best method by more than 10% when ten classes are learnt continually. Also, all of the above-mentioned methods are replay based approaches, while KCCIOL does not use any replay/memory.

9 Conclusion

We propose a novel approach to learn robust feature representations that are generalizable across future learning classes. Our approach uses a meta-learning framework with knowledge consolidation for learning generic feature representations that can be rapidly adapted for future classes without forgetting the previous classes under online updates to facilitate future learning. Our approach shows a significant improvement for class incremental online learning on several benchmark datasets.

Acknowledgements

PR thanks support from Qualcomm Innovation Fellowship and Visvesvaraya Young Faculty Fellowship.