Closed-Loop Memory GAN for Continual Learning

1 Introduction

Since early development and throughout life humans are constantly faced with unknowns in the environment which demand a persistent adaptation and expansion of past knowledge. In addition, as knowledge is expanded, learning is often facilitated since objects and tasks are often closely related and interconnected. For instance, during development, infants learn to categorize animals according to dimensions such as size, texture, shape, sound, among others. However, subsequent addition of new species rarely corrupts classification performance on the already learned categories. In fact, learning broad-species domains can aid in finer species discriminatory capability [Flesch et al., 2018].

/Contact Author

Nonetheless, recreating human-like lifelong continual learning remains a central challenge in Artificial Intelligence. State of the art deep neural networks (DNN) trained to perform supervised continual learning are known to undergo a phenomenon termed "catastrophic forgetting", which describes a sharp decline in the performance of the model on previously learned tasks as soon as a new task is introduced [French, 1999; McCloskey and Cohen, 1989]. This behavior does not come as a surprise if one recalls that in DNNs, learning an input output mapping implies parameterizing the network with an optimal weight set, through loss minimization. Thus, if training data is unavailable for previous tasks, there will be no more loss term for the old data and a weight parametrization may blatantly deviate from the previous optimal state incurring severe memory erasure.

2 PriorWork

In the recent literature, several methods have been proposed aiming to ameliorate catastrophic forgetting. They can be roughly subdivided into 3 groups: regularization, networkgrowing and replay approaches. With regularization methods, one constrains the change of learnable parameters to prevent "overwriting" what was previously encoded. For instance, [Li and Hoiem, 2017] perform distillation between multiple realizations of a network at distinct time-points, ensuring that the new weights do not shift significantly from the old. In a similar vein, [Kirkpatrick et al., 2017] operate within a single network model and use a Fisher information matrix computed with saved samples drawn from past tasks, which then acts as a regularizer preserving highly correlated weights. Similarly, [Zenke et al., 2017] use path integrals of loss-derivatives to constrain weights crucial to past tasks, yielding an intermediate parameterization with minimal combined loss.

Alternatively, in region-growing algorithms, the architecture itself is altered to accommodate new tasks followed by retraining. For instance, [Fernando et al., 2017] freeze the most important paths in the network, therefore forcefully preventing forgetting, and incrementally add new network chunks to incorporate new tasks. Lastly, In replay methods, the models no longer preserve a key pathway or weights. In these algorithms, one estimates the distribution of the old data either by saving a small fraction of the original dataset into a memory buffer or by training a generator to mimic the lost data and labels. At each new task, these methods learn by presenting a network with both new images as well as replay of estimated or buffered old images, reverting the continual framework into a multi-task setting and thus alleviating forgetting [Shin et al., 2017]. Other works have built on the idea of using a buffer of real data to approximate the past distribution [Rebuffi et al., 2017; Lopez-paz and Ranzato, 2017; Nguyen et al., 2018].

Yet, despite a growing number of appealing solutions, catastrophic forgetting is not a solved issue. Regularization methods have been shown to perform poorly in single-headed incremental class learning, for instance [Kemker and Kanan, 2018; Parisi et al., 2019], and here we reproduce this limitation in our own results for elastic weight consolidation [Kirkpatrick et al., 2017]. On the other hand, region growing approaches, while usually providing a clean solution for constrained incremental problems, can quickly become memory expensive since they require both an architectural expansion and the storage of at least a portion of old data for retraining.

Likewise, replay methods also run into scalability issues. So far, generative replay models learn a data distribution by resorting to intermediate copy states of the generator. In Deep Generative Replay (DGR) an unconditional GAN is trained at each task to cumulatively generate and discriminate images. Since the proposed GAN is unconditional, they employ an additional classifier (Solver) which is trained in parallel to classify the generated images and assign corresponding labels [Shin et al., 2017]. During each task switch, DGR makes a copy of the generator and classifier networks and uses them to generate sample images and labels for the old tasks. In Memory Replay GAN (MeRGAN) with joint replay, [Wu et al., 2018] propose a modification in the DGR framework by substituting the unconditional GAN for an ACGAN, thereby eliminating the need for the additional solver. Copy operations are both expensive and often lead to image quality being degraded through consecutive tasks. Moreover, replicating network states successively is not a fully desirable solution since, from the biological perspective, a human brain cannot produce an "intermediate copy" of itself to transfer knowledge. Lastly, methods which rely rather on small subsets of past data, memory buffers, have shown to yield good results but they do not make explicit how much of the performance is due to the algorithm developed and how much is intrinsically due to the variability included in the buffer unit.

3 Closed Loop memory GAN

3.1 Model Overview

In this paper, we propose a hybrid approach between memory buffers and deep generative models aiming to specifically reduce memory costs and maximize both the classification performance and generated image quality throughout training. In our model, there is only one generator and embedded classifier trained cumulatively, with no intermediate copy step. In this framework, as a new task is learned, the old data is approximated by continuously sampling from the generator at its present state, forming a closed loop training paradigm.

Of course, since a new task also modifies the parameterization of the generator, this procedure cannot be applied without some verification that the generated images are reasonable approximations of the old distribution that has been lost. Our method tackles this issue by, first, using an image filtering step in which either the classifier or the discriminator is used to assess the sample image quality and, as a result, blocking bad images from entering the training loop. Second, we employ external regularization by constructing a small dynamic memory buffer with real data samples chosen to maximize image heterogeneity and to enforce smoothness in the representation of old classes. The image buffer has fixed memory allotment. Therefore, it is not allowed to grow which requires eliminating some old images to make room for new ones. The sampling for the old data is then always a combination of buffer samples and "on-the-fly" generated samples, which provide a stochastic up-sampling of the memory unit.

3.2 Model Architecture

A vanilla GAN consists of two networks, a Generator and a Discriminator, competing with each other in a zero-sum game framework. The core block of our model (CloGAN), see figure (1), is a modified GAN termed Auxiliary Conditional Generative Adversarial (AC-GAN) [Odena et al., 2017]. The AC-GAN is also composed of 2 networks, but it includes a classifier combined in the same architecture as the discriminator, via an expansion to K+1 output nodes, for K classes plus the original vanilla Real/Fake discriminator output.

In an AC-GAN framework the generator is fed a uniform noise z ≈pz appended with a corresponding class label c ≈pc. Thus, the conditional generator, described by �G, generates an image x ∅G�G (z, c) and the AC-GAN learns a mapping in which the noise z is independent of the class c, enabling multiple class outputs for a fixed noise input. While the generator �G is trained to generate images as closely resembling the input image distribution, the discriminator, �D, is conversely trained to discriminate these generated images as fake, loss LFT. The embedded classifier, �C, shares most weights with the discriminator and generates a label prediction which, if incorrect, contributes to the overall loss of both generator and discriminator, LC. Overall, an AC-GAN is easier to train than a conventional vanilla GAN while also producing higher quality images. The loss functions are given as follows in (1) and (4) for generator and discriminator/classifier respectively.

Note that a plausible alternative to using a GAN would be to use a variational auto encoder (VAE) instead [Kingma et al., ]. However, in our testing, we have not been able to achieve results with a VAE as good as those presented here using a GAN. Hence, in the following, we restrict our analysis to approaches based on GAN. Details of the implementation can be found in the supplementary materials link.

3.3 Closed-Loop Training with Replay

In the continual learning setting, our method approximates the likelihood of old data by employing CloGAN to continuously output fresh new images at each mini-batch during training. A combination of image filtering and external regularization by an image memory buffer confer stability to the closed-loop procedure. At each task, our model is trained using an extended dataset which includes real images for the new task, GAN replayed images for old tasks, and memory images, forming an extended training set St (8), see figure 2. The memory component can be given a weighted importance, ,mem. The network is then trained by minimizing (9,10).

3.4 Image Filtering

At each mini-batch, the generator outputs fresh images approximating samples from old tasks, with the intent of producing a stochastic up-sampling of the reduced memory core. However, since these images are then used as training data in a closed loop, they have to be of the best quality possible to minimize error propagation. Thus, at each generation step, images are assessed for their quality and "filtered" out if they do not correspond to the standard.

Here, we use the embedded classifier in CloGAN to generate a prediction for the conditional image. If this prediction does not match the conditioning label, the image is filtered out. When old images are generated for closed-loop replay, they are sampled from a model which has already previously

Algorithm 1: CloGAN Train

Input : Data S real t , ..., S r eal T ; Require: T: Number of Tasks; It : Number of iterations; B : Buffer Size; Kc : Number of clusters per class; ,mem : Memory importance; 1 �/G ,�/D,C ,�/C ℘Train AC-GAN(St∅1) for i ∅1 to I1 2 S memory t∅1 ℘BufferConstruct (K,B,S real t ) 3 for t℘2 to T do 4 S/t ∅S real t [,memS memory tϒ1 5 for i℘1 to It do 6 S i/ t ℘Batch(S/t ) 7 S GAN tϒ1 ℘Forward (G(z, y c tϒ1 )) 8 S GAN tϒ1 ℘Filter (S GAN tϒ1 ) 9 S i t ℘ S i/ t [S GAN tϒ1 10 �/G , �/D,C , �/C ℘ Train AC-GAN(S i t ) 11 end 12 S memory t ℘BufferConstruct (K,B,S memory tϒ1 , S r eal t ) 13 end

converged for generation and classification of old tasks. The rationale behind this evaluation is that images which are missclassified have a higher probability of being distorted because of the ongoing training of the new task, and of deviating too grossly from the original distribution. We term this method Class-Conditioned Filtering (CFM).

In addition to CFM, we implemented a more complex procedure, "Discriminator Rejection Sampling" (DRS) proposed in [Azadi et al., 2018]. The latter employs the discriminator of an AC-GAN to approximately correct errors in the GAN generated distribution. Details of the implementation can be found in the supplementary materials link. We compare both to a baseline case for DRS which rejects a sample if the output from its discriminator logit layer has a score below some threshold, Soft rejection Filtering (SRF) [MacKay, 2003]. Overall, we found that CFM, DRS and SFR perform equivalently well. A table with comparisons is included in the supplementary materials link. Hence, since CFM has a much faster running time, we opted for carrying out only class conditional filtering in our final model.

3.5 Dynamic Memory Buffer

We fill a small memory buffer with samples and labels of original past data to perform external regularization. The memory can be seen as a stable reference frame throughout training that enforces a "smoothness" in the representation for each class. At each task, a selection method is employed to choose the samples from the new task which will go into the buffer, with the aim to maximize sample heterogeneity. Also, since a buffer has fixed size, this selection method is further used to determine which of the old task samples will be removed to make space for the incoming new data, employing again the heuristics of sample heterogeneity. Several buffer selection strategies were initially experimented but the best selection scheme was K-means clustering per class, both at image insertion and removal. In more details, the construction scheme is as follows: at the end of each current task, a k-centers algorithm is run per each class in the current tasks's training labels, super-labeling each image as one of K clusters. At the time of insertion into the memory buffer, we select equal numbers of image samples from each class-specific cluster. Additionally, if the buffer is full we compute the space needed for new images and remove an equivalent number of old images. We do this by assessing their stored super-cluster labels and removing equal amounts of samples per cluster, thereby preserving heterogeneity. By storing the per-class, cluster assignment superlabels we also avoid repeating the clustering operation.

3.6 Continual Learning Baselines

We evaluate other continual learning algorithms as baseline comparisons. We implement Elastic Weight Consolidation (EWC; Kirkpatrick [2017]), Deep Generative Replay (DGR; Shin [2017]) and Memory Replay GAN (MeRGAN; Wu [2018]) . With DGR, to make our implementation a fair comparison, we use an unconditional GAN with the same architecture and complexity as our CloGAN, except that it has only one Real/Fake output node. For both EWC and DGR, we use a classifier with identical architecture as our embedded classifier/discriminator, but with one fewer output node since a pure classifier does not evaluate Real/Fake attribution. Finally, for MeRGAN we implement an AC-GAN with identical architecture as our CloGAN.

4 Experiments

4.1 Buffer Selection

We experimented with several buffer selection schemes but they under-performed class-specific K-centers. In the other selection methods, we extracted the logit or softmax layer of the discriminator/classifier network and computed measures such as Kurtosis and Peak-Difference to assess sample heterogeneity. The latter measure corresponds to the difference between softmax scores of the most probable and second most probable class for a given image. As such, we ranked the images according to each measure and kept the images with a probability proportional to their score. In other words, we performed a roulette weighting procedure such as in genetic selection [Goldberg and Deb, 1991]. Table 1 contains performance metrics for 3 buffer selection schemes and no selection (none) during CloGAN incremental class learning using the FASHION dataset with memory buffer of size 0.16%.

4.2 Incremental Learning

We evaluate continual learning as accumulating knowledge of a growing number of disjoint classes, termed incremental learning. Furthermore, we make use of a challenging variation of incremental learning, "single headed learning". Here, each task is a disjoint subset of classes from the overall dataset. Performance is evaluated for all previous classes, resulting in a 1/K chance level, where K is the number of classes accumulated to that point. We evaluate incremental class learning in 4 datasets: MNIST [LeCun et al., ], FASHION [Xiao et al., 2017], SVHN [Netzer et al., 2011] and EMNIST [Cohen et al., 2017]. The first 3 were subdivided in disjoint subsets of 2 classes per task, with a total of 5 tasks to cover all the label types. E-MNIST, a larger dataset, was divided into tasks of 3 classes, covering 24 different classes in 8 consecutive tasks. To account for the growing number of classes, we create extra output nodes which are incrementally used, which allows us a single head for all tasks.

We distinguish our procedure from Multi-Headed learning [Farquhar and Gal, 2018] in which prediction is constrained to classes in each task. For instance, a multi-headed version of our MNIST test would use and re-use only two output nodes. After training on full disjoint MNIST with 5 tasks of 2 classes each, when evaluating the first task (digits 0 and 1), a multi-headed would only have to decide between digit 0 vs 1, as opposed to a one in ten decision for single-headed. This typically leads to much higher accuracies partially because an output node never becomes completely disabled, as it is always used for the last task. Finally note that a multiheaded network with only 2 output nodes provides an output that needs to be further disambiguated by knowing the task.

Average Continual Performance

Figure 3 displays the average performance of CloGAN when varying memory buffer size. Our method avoids catastrophic forgetting even with very small buffer sizes such as 0.08% (50 images) and 0.16% (100 images), for both MNIST and FASHION. For the more challenging E-MNIST and SVHN, buffer requirement becomes more demanding. Nonetheless, we obtain superior performance over the competing methods with still very reduced memory sizes: only 0.5% (576 images) and 1% (492 images).

Table 2 compares maximum average accuracies after training all tasks, for all methods tested. First, when no memory or GAN sampling is performed, the "FGD" condition which contains only fine-tuning with gradient descent, catastrophic forgetting occurs. Second, EWC accuracy rapidly declines, asymptotically reaching the catastrophic forgetting curve. EWC has already been shown to behave poorly in incremental single-headed paradigms [Kemker and Kanan, 2018; Farquhar and Gal, 2018]. To further confirm that this degradation of performance was not particular to our implementation, we replicated the permuted-MNIST experiment proposed in the original EWC paper ; and verified that in this learning paradigm EWC performs very well. This discrepancy between the experiments is likely due to the difference in output mapping, see supplementary.

Lastly, we report the accuracies for the deep replay methods, DGR and MeRGAN. For MNIST, both DGR and MeR- GAN perform very well, reaching 94.9 % and 98.25% whereas CloGAN achieves accuracies of 92.26% with memory of 0.16% and 98.03 with (1.6%). However, for all other datasets, which are significantly harder than MNIST, DGR and MeRGAN both underperform CloGAN by significant amounts. For SVHN, the most challenging, both DGR and MeRGAN display degraded performance after the first task. This behavior likely has cause in a persistent degradation of generated image quality throughout training. Both methods represent old data exclusively by replayed images from an intermediate generator copy. If the generator cannot produce images which represent the original distribution with high fidelity, the gap in representation capacity can be enlarged and propagated through successive GAN transfer (copy) operations. CloGAN alleviates GAN representation degeneration because it is trained from an extended set containing both replay images from the generator and real images in the buffer. The real images never degenerate and act as an anchor to keep smoothness and quality in the subsequent generated images. DGR has another disadvantage over CloGAN: it does not generate conditioned images, requiring a separate classifier to produce old image labels during training. If that classifier does not have perfect performance, it will inevitably misslabel some images, contributing to error propagation. In the supplementary we tested a new copy-CloGAN that copies the generator at each task switch. However, the copy operation did not provide an advantage if comparing memory usage.

Stochastic Up-Sampling

We confirm that CloGAN performs an upsampling of the memory buffer selection by comparing our method to two variations in which the AC-GAN is trained only from a memory buffer, both in continual (Frozen-CloGAN) and multitask settings (MT). For the latter two conditions there is no closed-loop replay of GAN samples. Furthermore, in the MT setting we re-start training at each task switch. The results reported in figure 4 correspond to the maximum accuracies achieved for each task for all 3 variations. We verify that stochastic generation in CloGAN provides an upsampling of the buffer and achieves superior performance to FrozenCloGAN and MT. We show results for the more challenging datasets, E-MNIST and SVHN.

Upsampling is indicated when a positive gap between CloGAN and Frozen-CloGAN increases as more tasks are added. For SVHN, the last task shows clear gaps between CloGAN Frozen-CloGAN as well as MT (maximum gap of 11.39% at task 5). Similarly, E-MNIST shows a clear gap in the last two tasks, 7 and 8 (maximum gap of 10.41% at task 8). Additionally, we show that MT under-performs starting in early tasks due to lack of forward transfer since the networks are re-started from scratch at each task switch. Similar upsampling behavior was observed in MNIST and FASHION, with maximum gaps of 6.84% and 9.35% respectively. Additional figures can be found in the supplementary link.

Per Task Performance

In figure 5A,B), we exhibit per task accuracies along time. Here, CloGAN is shown to produce stable performance throughout consecutive tasks. For both E-MNIST and SVHN all past tasks maintain high accuracies consistently throughout learning of new classes. For example in EMNIST, task 1 preserves its accuracy at 84.33% despite the learning of 7 other tasks in succession. Likewise, SVHN task 1 has an accuracy of 83.87 %. The results are significantly higher when compared to the baseline of catastrophic forgetting and EWC. Moreover, we also display performance for MeRGAN. In EMNIST, MeRGAN accuracies for tasks 1 and 2 are clearly underperforming CloGAN at the end of training, likely due to image degradation from GAN to GAN transfer.

In figure 5C), we show generated images by CloGAN and MeRGAN for both SVHN and E-MNIST. We list images taken after training of all tasks. For SVHN we list all classes cumulatively learned. For E-MNIST, since there are 24 classes, we limit the display to the two first tasks as well as the last task (8th). For CloGAN, we find that images are sharp even when using small memory sizes, 1% - SHVN and 0.5% - EMNIST. This is true for beginning tasks as well as latter tasks. In contrast, in MeRGAN former taks are sharply more degenerated than latter ones. In EMNIST this can be seen by an overall darkening of letters a through f .

5 Conclusion

In conclusion, we have shown how using very small buffers in conjunction with stochastic replay can give rise to superior performance compared to simple gradient descent, EWC or other replay methods. In our model, CloGAN, the memory buffer acts as an external regularization for the generator, counteracting image degradation through time. Our approach is relatively easy to implement and necessitates only low computation (no full retraining) and memory (small buffer), making it ideal to enable life-long learning on resourceconstrained mobile (at the edge) devices.

Acknowledgments

This work was supported by the NSF (CCF-1317433, CNS1545089), C-BRIC (SRC - DARPA) and Intel Corporation.