Learn from Concepts: Towards the Purified Memory for Few-shot Learning

1 Introduction

The success of deep learning stems from a large amount of labeled data [Noh et al., 2017; Bertinetto et al., 2016; Long et al., 2015], while humans have a good generalization ability by only seeing a few number of samples. The gap between the two facts brings great attention to the research of few-shot learning [Vinyals et al., 2016; Finn et al., 2017; Sung et al., 2018]. Unlike traditional deep learning scenario, few-shot learning is not to classify unseen samples, but fast adapts the meta-knowledge to new tasks, where only very few labeled data and knowledge gained from previous experience are given.

Recently, significant advantages [Vinyals et al., 2016;

∗Equal Contribution

†Contact Author

Finn et al., 2017; Snell et al., 2017; Sung et al., 2018] have been made to tackle this problem by using the idea of meta-learning coupled with episodic training [Vinyals et al., 2016]. The intuition is to use a episodic sampling strategy, a promising trend to transfer knowledge from known categories (i.e., seen categories with sufficient training examples) to new categories (i.e., novel categories with few examples), simulating the human learning process. In this framework, metric-based approaches [Vinyals et al., 2016; Snell et al., 2017] and graph-based approaches [Garcia and Bruna, 2017; Liu et al., 2018; Kim et al., 2019; Yang et al., 2020] are two representative methods by taking the primary advantage of the transferable meta-knowledge. Due to the ability to learn from graph data efficiently, graph-based approaches generally outperform metric-based method, which extends the pairwise query-support relationship to a graph structure.

In spite of the effectiveness of graph-based approaches [Kim et al., 2019; Yang et al., 2020], most of them ignore a critical issue that how the knowledge learned in the past could be useful for the new task when the episodes are trained one by one. As a kind of intuition, for an unseen task, human beings don't use the whole knowledge, but a few informative and relevant concepts, to improve the prediction ability for the new task. For example, if a man has learned the concepts of horses, tigers and pandas, it is easy to recognize zebras by finding zebras have the outline like horses, stripes like tigers, and black and white color like pandas. Motivated by this simple intuition, we propose an assumption that a few-shot learning model should explicitly establish the relationship between episodes and take full use of existing learned knowledge.

However, it raises two fundamental problems hindering existing graph-based approaches: 1) how to learn a stable and consistent concepts when episodes are rapidly coming; 2) how the learned concepts further help the prediction when adapting to new tasks. In this paper, we propose a purified memory framework to tackle these two problems. Our basic idea is simply that simulated the recognition process of human beings. To keep stable and consistent concepts, we hold a memory bank during episodic training, which learns an optimal prototype representation for each category from the perspective of information bottleneck principle [Tishby and Zaslavsky, 2015]. By progressively purifying the information from semantic label, the stored knowledge is supposed to be generally expressive, consistent and stable.

To make full use of the purified memory, we present a Graph Augmentation Module (GAM) as a way of mining meta-knowledge and establishing the correlation between different episodes. When dealing with a new task, GAM first retrieves the concepts of k-nearest neighbors by taking class center from the current task as query. Then retrieved concepts and episodic training samples are forwarded into a graph neural network (GNNs) with an adaptive weighting scheme. Thus the concepts learned in the past and knowledge learned from the new task are aggregated, which allow our model to make accurate prediction. It is worth noting our method is a model-agnostic approach and could be integrated into any advanced GNNs method flexibly with negligible computational cost.

Our major contributions are three-fold: (1) we present a new memory purifying mechanism with efficiency, consistency and vigorous expressive power; (2) the proposed GAM is able to mine the meta-knowledge and capture the correlation between different episodes; (3) our approach yields stateof-the-art few-shot results and our intriguing findings highlight the need to rethink the way we use meta-knowledge.

2 Method

This paper aims to address the problem of few-shot classification. The problem definition is fundamentally different from traditional classification, whose objective is not classify unseen samples but to fast adapt the meta-knowledge to new tasks. Specifically, a labeled dataset with sufficient training samples from base classes C base is provided, and the goal is to learn the concepts with very limited data collected from a set of novel classes C novel , where C base ∩ C novel = ∅. An effective way to solve the few-shot problem is to use the episodic sampling strategy. In this framework, the samples in meta-training and meta-testing are not samples but episodes {T }, each of which contains N classes (ways) and K shot per class. In particular, for a N-way K-shot task, a support set S = {(xi, yi)}N×K i=1 and a query set Q = {(xi, yi)}N×K+T i=N×K+1 are sampled. Here, xi and yi ∈ {C1, · · · CN} are the i-th input data and is fromC base . In the meta-test, a test task is also sampled with the same sized episode from unseen categories C novel . The aim is to classify T unlabeled samples in query set into N classes correctly.

2.1 Overview of Framework

The framework of the proposed method is illustrated in Fig. 1. It mainly consists of three components, i.e., an encoder for discriminative feature extraction, a memory module for expressive meta-knowledge storage, a graph augmentation module for comprehensive inference. In general, our approach can be summarized into 3 stages (i.e., Pre-Train, Meta-Train, Meta-Test).

Phase-I Pre-Train. We follow a simple baseline [Chen et al., 2020]: learning a supervised representation on the metatraining set C base , followed a linear classifier on top of this representation. It has been shown that this pre-train stage is beneficial for the downstream few-shot task [Tian et al., 2020], and the trained feature extractor (e.g., ResNet-12[He et al., 2016]) and classifier are then used as the initialization of our encoder and memory bank, respectively.

Phase-II Meta-Train. We first extract the features of support and query samples as task-relevant embeddings V t . Then to facilitate fast adaption, our approach holds a memory bank to store the expressive representations of the support set. This memory bank is optimized with a new updating scheme to progressively purify discriminative information (introduced in Sec. 2.2). Further, the purified memory is incorporated with a graph augmentation module for robust prediction (introduced in Sec. 2.3). In this module, we mine the relevant prototypes V m, referred as meta-knowledge in this paper, to propagate the similarities between V t and V m via a graph neural network. Consequently, our model is able to generalize to new tasks conveniently with negligible memory cost.

Phase-III Meta-Test. The procedure of Meta-Test is similar with Meta-Train, where the episodic sampling strategy is also adopted. But unlike Phase-II, the memory bank and other modules will not be updated throughout the process. In other words, switch will be closed as shown in Figure 1.

2.2 RefinedMemory Updating

Meta-knowledge plays an import role in learning new concepts from unseen samples, and recent FSL advances [Ramalho and Garnelo, 2019] often exploit a memory mechanism to store this meta-knowledge. In its typical setting, the memory tries to preserve as much information as possible (e.g., store the whole features). However, we argue this strategy is both ineffective and inefficient. In the context of FSL, the episodic sampling makes the feature extractor rapidly learn new concept with very few samples, and this causes a problem that the feature in memory is updated when the feature extractor is under a very different task context. From this perspective, the representation learned from different tasks requires a purification process to be a stable concept.

To alleviate the above issues, we propose to refine the memory via learning an optimal prototype for each category. Specifically, considering a N-way K-shot task in FSL, we use f lsup ∈ R[N×K,d] to denote the feature representations of the support set in the l-th episode, and M ∈ R[C,d] to denote the memory bank, where C and d indicate the total number of categories and the dimension of the prototype, respectively.

To progressively purify semantic information from labels, we firstly conduct category-wise averaging to f lsup to obtain the centroids f l cen ∈ R[c,d] , each of which is then concatenated with the prototype f lp ∈ R[c,d] (stored in the memory) that belongs to the same category. We forward the concatenation f l cat ∈ R[c,2×d] to a fully-connected layer to reduce the dimension, and utilize the output to refine the memory. Here we propose to use the information bottleneck principle to purify the concept. The following constraint is used to ensure the IB to be well working, i.e., preserve semantic label information while avoiding task-irrelevant nuisances.

where I(.; .) denotes the mutual information, Y represents the label, and β is the Lagrange coefficient, respectively.

Specifically, Eq. (1) aims to learn prototype f lp that is maximally informative about the target Y while being maximally compressive about f l cat . However, Eq. (1) requires estimating mutual information in high dimension, which is intractable in such a high-dimension space. Fortunately, since our goal is to purify the concept, we show a self knowledge distillation loss could be strictly consistent with Eq. (1), where the mathematical deduction is shown in supplementary materials.

In practice, the following constraint is enforced to purify the discriminative information and further refine the memory:

Here, θ and φ denote the parameters of the encoder and the FC layer, DKL[.||.] represents the KL-divergence, y denotes the label. Note that both p(y|f l cat ) and p(y|f lp ) denote conditional distribution, and, in practice, are the output of an extra linear linear (detailed descriptions are shown in supplementary materials).

The refining of M is, in essence, iteratively aggregating discriminative information and diluting task-irrelevant nuisances. A naive solution is to append the output of the IB to M for every episode. But this solution leads to tremendous spatial and time cost and yields poor performance (see Sec. 3.4). In the view of above, we propose to refine the memory bank by momentum update. Formally, M is updated by:

where λ ∈ [0, 1) is a momentum coefficient and f lB ∈ Rd denotes the output of the IB at current episode.

In this way, the memory is supposed to be generally expressive, consistent and much more efficient. The refined prototype representations further incorporate and aggregate with meta-knowledge mining and is applied to facilitate the inference of FSL, as described next.

2.3 Graph AugmentationModule

For an unseen task, human beings don't use the whole knowledge, but a few informative and relevant concepts, to abstract the new task. Motivated by this, we propose a Metaknowledge Mining approach to simulate this behavior. The core idea behind our approach is to aggregate similar features, rather than the entire memory bank, to help our model learn new concepts for an unseen task. In particular, we use a graph augmentation module (GAM) to capture the relationship between a specific task context and relevant concepts. Their similarities are then propagated through a graph neural network [Kim et al., 2019], in which each layer performs node feature and edge feature update, to realize fast and comprehensive inference.

Meta-knowledge Mining. For each class centroid f l cen [i] in l-th episode, we first compute the cosine similarities between f l cen [i] and each prototype in the memory M. Then we select k-nearest-neighbors of f l cen [i], which are denoted as MK = {m1,m2, ...,mk}. In order to perform the aggregation, we use an attention coefficient calculated by the centroid f l cen [i] and selected embeddings mj :

where 〈·, ·〉 denotes the cosine similarity between two vectors and τ is a scalar parameter. Finally, the meta-knowledge node of each class is calculated as:

where [·; ·] is the concatenation operation and fagg(·; θagg) performs a transformation : R2d → Rd on the concatenated features which is composed of a fully-connected layer, with the parameter set θagg.

Augmented Graph Initialization. For a N-way K-Shot task, given the features extracted from the encoder and the mined meta-knowledge, a fully connected graph G = (V, E) is constructed, where V = {vt i }N×K+T i=1 ∪ {vm i }Ni=1 =

{vi}N×(K+1) i=1 and E = {eij}i,j=1,...,|V | denote the set of nodes and edges, respectively. The node contains two types of points i.e., task-relevant nodes V t and meta-knowledge nodes V m. The edge represents the similarity between two nodes and is initialized as:

where ˆ S = S ∪ V m denotes the union of support set and augmented meta-knowledge. Thus the meta-knowledge is augmented to existing inference task and allow the model to adapt to new task by taking advantage of learned concepts.

Node Feature Update. Given vℓ-1 i and eℓ-1 ij from layer ℓ-1, the feature node vℓ i at layer ℓ is updated by a neighborhood aggregation procedure. This aggregation is weighted by the edge similarity between two neighbors. A feature transformation is also conducted to normalize the feature. Mathematically, the node feature update is defined as:

where [·; ·] is the concatenation operation and fnode(·; θnode) is a transformation block consisting of two convolutional layers[Glorot et al., 2011; Ioffe and Szegedy, 2015], a LeakyReLU activation and a dropout layer.

Edge Feature Update. Edge feature update is done based on the newly updated node features vℓ i . The similarities between every pair of nodes are re-calculated, and the feature of each edge eℓij is updated by combining the previous edge feature value eℓ-1 ij and the updated similarities as:

where fedge(·; θedge) is a metric network parameterized by θedge, which includes four convolutional blocks, a batch normalization layer, a LeakyReLu activation and a dropout layer. It is worth noting our GAM can be implemented with any other GNN, and substantially improve their performance.

2.4 Prediction and Optimization

When the optimization is complicated, the predicted probability of a node vi belonging to Ck can be denoted as:

where δ(yj = Ck) is the Kronecker delta function that is equal to one when yj = Ck and zero otherwise, eij denotes the edge feature between two nodes vi and vj. A softmax layer is then used to normalize this probability.

During the meta-training stage, our model is optimized by minimizing the binary cross-entropy loss (BCE):

where ei and ˆ yℓ i are the ground-truth of query node edge-label and the query-edge predictions, respectively, and λℓ is the coefficient for ℓ-th layer. In order to make the meta-knowledge nodes consistent with the predicted label, we also introduce another binary cross-entropy loss (BCE) Lm to estimate the discrepancy between the ground-truth and the predictions of meta-knowledge nodes edge-label.

Finally, the total loss L can be defined as:

where α is the coefficient to balance Lq and Lm. In our experiments, we fix α = 0.2 and β = 0.01.

3 Experiments

3.1 Experimental Setup

Datasets. We evaluate our approach on four few-shot learning benchmarks followed by [Yang et al., 2020]: miniImageNet [Vinyals et al., 2016], tieredImageNet [Ren et al., 2018], CUB-200-2011[Wah et al., 2011] and CIFAR-FS [Bertinetto et al., 2018]. Among them, miniImageNet and tieredImageNet are collected from ImageNet, and CIFAR-FS is a subset from CIFAR-100. Unlike these datasets, CUB200-2011 is a fine-grained bird classification dataset.

Evaluation. For evaluation, all the results are obtained under standard few-shot classification protocol: 5-way 1-shot and 5-shot task. No matter in 1 or 5-shot setting, only 1 query sample each class is used to test the accuracy. We report the mean accuracy (%) of 10K randomly generated episodes as well as the 95% intervals on test set. Notice that all the hyperparameters are determined from the validation sets.

3.2 Implementation Details

Network Architecture. We utilize two networks as our encoder backbone,(i.e., ConvNet and Resnet12 [Kim et al., 2019; Lee et al., 2019]). ConvNet contains four blocks, and each block includes a 3x3 convolutional layer, a batch normalization layer and a LeakyReLU activation. Similarly, ResNet12 consists of four residual blocks. Please refer to [He et al., 2016] a comprehensive understanding. After the backbone network, there is a global average pooling layer and a fully-connected layer to produce 128-dimensional instance embeddings.

Training. In the pre-training stage, the baseline following prior work [Chen et al., 2020] is trained from scratch with a batch size of 128 by minimizing the standard crossentropy loss on base classes. After that, we randomly select 40 episodes per iteration for training the ConvNet in the meta-train stage. This sampling strategy is slightly different for ResNet12, where 5-way 5-shot task, we only sample 20 episodes per iteration due to the memory cost. The Adam optimizer is used in all experiments with the initial learning rate of 10-3 . We decay the learning rate by 0.1 per 8000 iterations and set the weight dacay to 10-5 . We train 50,000 epochs in total, and the encoder are frozen for the first 25000 iterations.

3.3 Main Results

In this section, we demonstrate the effectiveness of our approach against state-of-the-arts methods. For a fair comparison, we adopt two representative few-shot graph neural networks i.e., EGNN and DPGN as our GAMmodule. Further, by using two kinds of backbone ConvNet and ResNet12, we report the results under 5-way 1-shot and 5-way 1-shot setting on all benchmark datasets for a comprehensive evaluation.

Results on Generic Object Recognition. For the generic object classification, we evaluate our approach on miniImageNet, tieredImageNet and CIFAR-FS and report the results in Table 1. The main observations are as follows: 1) The proposed method outperforms all competitors demonstrating the effectiveness of our method. Further, the performance obtained by using ResNet12 is superior to that using ConvNet due to better representational capacity. 2) No matter which graph neural network are used, the proposed method evidently outperform the baseline with a clear margin. 3) In both 1-shot or 5-shot setting, our method is basically stable at the top performance. The improvement is more significant under 1-shot setting due to the purified memory. Hence it appears that our methods will be more effective when facing new tasks with fewer samples.

Result on Fine-grainedClassification. For the fine-grained bird classification problem, the results of CUB-200-2011 is reported in Table 1. In particular, our method also outperforms other competitors by a large margin. Notice that on this dataset, different graph neural networks and backbones have less impact on the performance.

Discussion. Since the proposed method is based on the GNNs framework, Our approach could be integrated into any advanced GNNs method flexibly. Our results show that with purified memory and GAM module, the performance of GNNs would be remarkably promoted.

3.4 Ablation Study

We present experiments to confirm our main claims: 1) Purified memory can facilitate fast adaption. 2) Meta-knowledge with GAM is able to promote the existing GNNs models. All the experiments are conducted on tieredImageNet under 5way 1-shot setting with ResNet12. Also the quantitative results of 5-shot are shown in supplementary materials.

Impact of Purified Memory. We compare four different memory banks and the results are illustrated in Fig. 2. Note that the baseline degenerates to EGNN when without memory. We can draw the following conclusions: 1) Without the help of memory, GAM can boost the performance of GNNs model, even incorporating with the class center in the current episode. 2) Three different memory banks are evidently superior to Non-Mem baseline, showing the importance of metaknowledge. 3) The prototype-based memory is more efficient and effective, which confirms our assumption that storing the whole features is a sub-optimal solution. 4) The experimental results support our motivation of obtaining an optimal prototype representation for each category. Meanwhile, it seems that the memory cost of the proposed method is keeping the same level compared with the baseline.

Impact of GAM. To demonstrate the effectiveness of our graph augmented module, we first visualize the embedding space in Fig 3. In particular, we randomly select 5 classes, each of which contains 200 samples from tieredImageNet. We project the features trained by EGNN and our EGNN equipped with GAM into a 2D plane via t-SNE. The results show that the embedding space is mixed in EGNN, such that the discriminative ability of learned model is naturally limited. On the contrary, our model is able to distinguish different categories with a large inter-class margin, hence we get a substantial improvement. This indicates that with the help of purified meta-knowledge, the discriminative information could be further highlighted via the GAM.

Additionally, to visualize how the meta-knowledge help the prediction process, we choose a test scenario where the ground truth classes of five query images are non-overlapping (i.e., 5-way 1-shot) and visualize instance-level similarities as shown in Figure 5. Specifically, we select two kinds of instance-level similarities to demonstrate the effectiveness of our approach. Notably, the heatmap shows GAM refines the instance-level similarity matrix after several layers and makes the right predictions for five query samples in the final layer compared with EGNN. We can also find this refinement is owing to the augmentation of meta-knowledge node. Due to the purified concept, the heatmap is essentially clean and hence the meta-knowledge provides auxiliary strong supervision. These similarities are then further propagated through the graph neural network, allowing the model to take the advantages of memorized concept and knowledge learned from new tasks. This experimental result convincingly supports our hypothesis.

Impact of k-nearest neighbor. In theMeta-knowledgeMining stage, we retrieve the most similar k samples from memory to augment the graph. Here we discuss its impact when the k varies. As shown in Figure 4, few-shot recognition performance keeps improving when k increases, and when k increases to a certain value, the accuracy begins to decline on both datasets. Hence it s recommended to set this value as 6, empirically.

4 Conclusion

In this work, we have presented a new memory updating scheme for few-shot learning, which progressively purifies the semantic label information from the perspective of information theory. Purified memory is generally expressive, consistent, efficient, and then naturally cooperated with a graph augmented module. GAM further exploits the metaknowledge and the knowledge learned from the new task to make a precise prediction. This scheme is a model-agnostic module and could be integrated into any advanced GNNs method flexibly.

Acknowledgements

This work is sponsored by National Natural Science Foundation of China (61972157, 61772524, 61876161, 61902129, 61976212), Natural Science Foundation of Shanghai(20ZR1417700), CAAI-Huawei MindSpore Open Fund, Shanghai Sailing Program (21YF1411200), National Key Research and Development Program of China (2019YFC1521104), Zhejiang Lab (No.2019KD0AC02, 2020NB0AB01), the Fundamental Research Funds for the Central Universities.