Learning Task-aware Local Representations for Few-shot Learning

1 Introduction

Deep learning based methods [Krizhevsky et al., 2012; He et al., 2016] have achieved state-of-the-art performance on a variety of visual recognition tasks. These supervised methods need a lot of labeled data with diverse visual variations to effectively train a network. However, collecting a large amount of labeled data is time-consuming and laborious. In contrast, humans can recognize classes with extremely few labeled examples. Therefore, for machine learning algorithms, how to recognize classes with extremely few labeled examples, i.e., few-shot learning, has attracted a lot of interests. Few-shot learning attempts to transfer the knowledge like humans, for generalizing to novel classes with very few supervisions.

∗Contact Author

To address few-shot learning tasks, a lot of methods have been proposed. However, most of these methods [Vinyals et al., 2016; Snell et al., 2017] adopt an image-level feature for classification and make an assumption that the image-level deep embedding space for the seen classes is extensively effective for the unseen classes, which is somewhat idealistic in practice. Fortunately, although the image-level embedding space is not equally effective for the seen and unseen classes, the low-level information, i.e., the local representations (LRs) of semantic patches, among the seen and unseen classes, generally remain similar. Some recent methods [Li et al., 2019c; Li et al., 2019b; Sung et al., 2018] have taken feature representations of semantic patches (i.e., LRs) into consideration, but they do not fully exploit the capabilities of LRs in the context of the entire task. Recall the way that humans recognize an instance (object) into one of several unseen classes. It is quite natural that he/she will look for the distinct semantic patches which are only shared between the certain class and the query image. In other words, the semantic patches commonly shared by all classes are not truly important for recognizing a novel instance. For example, the way we recognize a "bird" among the "dog" and "cat" is quite different from the one among the "airplane" and "dragonfly". For the former one, the wings are important but are not the key concern for the latter one. Similarly, the fur and feather are more important for the latter one than the former. In other words, the importance of the semantic patches changes with the tasks.

As described above, the existing LRs based few-shot methods have not yet made full use of the information provided by the LRs mainly in two aspects: (1) the LRs are only considered inside one image or one class individually (i.e., imageto-image or image-to-class manner), rather than the entire task; (2) the semantic local patches are weighted equally, rather than the more discriminative patches enjoy the higher weights. To overcome these two limitations, we design a episodic attention mechanism, which can select and weight the key patches without paying too much attention to the common parts among the entire task. Note that, in the work of [Li et al., 2019b], a rank-based selection, i.e., k-nearest neighbor (k-NN) selection, is utilized to select k (e.g., k = 3) most related patches in each class for a query local instance. However, the number of related semantic patches for a query local instance shall be dynamically changed according to the context of the current task. It means that we may need more discriminative patches to recognize an object in one task, but just need much fewer patches to recognize the same object in another task. By contrast, the value range of the relationship between the related semantic patches mainly depends on semantic patches' nature, and remains relatively stable in changing tasks, so that we propose a value-based selection with a threshold to replace the rank-based selection. And then, we name the network with the above episodic attention mechanism as Task-aware Local Representations Network (TL-Net).

Although the above mentioned value-based selection is appealing, it roughly sets a global manual threshold for all the semantic patches of tasks, which is hard to be effective for different semantic patches at the same time. To this end, we develop a trainable module to adaptively learn this threshold for each semantic patch, i.e., adaptive value-based selection. In this way, for each certain semantic patch, we can obtain its own relation threshold according to its nature. Typically, we call the extended TL-Net with learnable thresholds as ATLNet, Adaptive Task-aware Local Representations Network, to show the additional adaptive ability relative to TL-Net.

Our contributions can be summarized as follows:

• We propose a novel episodic attention mechanism by exploring and weighting discriminative semantic patches inside the entire task, aiming to learn task-aware local representations for few-shot learning. Moreover, instead of the rank-based selection, a feasible value-based selection strategy is proposed.
• We further develop a trainable module to design an adaptive value-based selection strategy, making it possible to dynamically and adaptively select discriminative semantic patches for different tasks.
• We conduct comprehensive experiments on the challenging miniImagenet and three fine-grained datasets to verify that the proposed ATL-Net achieves superior performance over the state-of-the-art methods.

2 RelatedWork

The recent literature of few-shot learning mainly comes from the following two categories: meta-learning based methods and metric-learning based methods.

2.1 Meta-learning basedMethods

Meta-learning based methods learn the learning algorithm itself. [Santoro et al., 2016] proposes an LSTM-based metalearner to interact with an external memory module. The proposed framework in [Santoro et al., 2016] adopts an LSTMbased meta-learner to learn a distinct optimization algorithm to train a classifier as well as learning a task-aware initialization for this classifier. MAML and its variants [Finn et al., 2017] train a meta-leaner to provide suitable parameter initialization, so that they can be quickly adapted to a novel task. Similarly, [Li et al., 2017] adjusts the update direction and learning rate for quickly adapting to a novel task. [Cai et al., 2018] introduces the memory slots to construct a contextual learner for predicting the parameters of an embedding module for unlabeled images.

Nevertheless, these methods often need costly higher-order gradients or need another complicated memory structure, making these methods difficult to train and may lead to failure when scaling to deeper network architectures [Mishra et al., 2018]. Compared with methods in this branch, the proposed ATL-Net can achieve competitive results with a much simpler network architecture, which is trained from scratch without fine-tuning.

2.2 Metric-learning basedMethods

Metric-learning based methods address the few-shot classification problem by "learning to compare". [Koch et al., 2015] proposes a Siamese neural network to learn generic image representations, which is conducted as a binary classification network and trained by a regularized cross-entropy loss. [Vinyals et al., 2016] introduces an episodic training mechanism into few-shot learning and proposes the Matching Net by using attention and memory together. [Snell et al., 2017] proposes a Prototypical Net by measuring the Euclidean distance between the class-mean feature and the query feature.

However, the above methods usually adopt an image-level global feature to represent each image based on a somewhat ideal assumption that the seen and unseen classes sharing a relatively consistent embedding space. In contrast, the lowlevel information, i.e., the local representations (LRs) of semantic patches, is more consistent and transferable than the high-level global features among the seen and unseen classes, which has been verified in some recent work. For example, [Sung et al., 2018] measures the distances between the query images and the support images by applying convolution layers on the concatenated feature maps, which implicitly uses the LRs. [Li et al., 2019b] proposes DN4 to explicitly utilize the LRs through a k-nearest neighbor selection and enlarges the image-to-image search space to a more effective imageto-class one. However, these methods only consider the relationship between query images and classes at an image-level or a class-level without adequately mining the important information hidden behind the LRs at the task-level.

Different from the methods above, our ATL-Net can explore richer information of the LRs at the task-level and can adaptively select the key semantic patches for a specific task, as the progress of the human beings. Experiments on the challenging general and fine-grained datasets show the superiority of our method compared with other state-of-the-art methods.

3 The ProposedMethod

3.1 ProblemDefinition

In this paper, we follow the common settings of few-shot learning methods. Given a small support set S which consists of N unseen classes with K samples per class, our goal is to classify a query sample q ∈ Q into one of the N support classes, which is called an N-way K-shot task. To achieve this goal, an auxiliary set A is employed to learn transferable knowledge using the episodic training mechanism [Vinyals et al., 2016]. We divide A into many N-way K-shot tasks {T }, where each T contains an auxiliary support set AS and an auxiliary query set AQ. In the training stage, hundreds of tasks are fed into the model, encouraging the model to learn Support Set Query Image Feature Maps transferable knowledge that can be used in new N-way Kshot tasks (i.e., S and Q) with unseen classes. Note that S and A own different label spaces with no intersection.

Figure 1: The overview of the proposed method under 5-way 1-shot setting. The model mainly consists of two parts: the embedding module FΘ to learn local representations and the adaptive episodic attention module FA to generate adaptive episodic attention for selecting discriminative patches for a special task. The score with the red circle indicates the predicted label. (Best viewed in color.)

Our overall framework is illustrated in Figure 1. All the images are first embedded into feature representations by an embedding module FΘ. A local relation map MR is then calculated to capture the local relationship between the query image and the support set. Meanwhile, the adaptive episodic attention module FA learns an episodic attention map MA, which can adaptively select the discriminative local patches among the support set for a certain query patch, as the process of human recognition. Note that the episodic attention focuses on the relations between the local patches, not isolated individuals. After that, we apply the attention mapMA onto the relation matrix MR through an element-wise multiplication to eliminate noise, i.e., the relation constructed by the commonly shared patches among the task, and then enhance the discriminative information. Finally, we can directly get the final score for classification from the processed relation matrix through naive methods, like addition.

3.2 Task-aware Local Representations

Let x ∈ S ∪ Q denote an input image, we first feed it into the embedding module FΘ to obtain a feature representation FΘ(x) ∈ RC×H×W . Typically, we can get HW C-dimensional LRs for each input image, making up a total number of NKHW support LRs, i.e., LS = FΘ(S) ∈ RC×NKHW and HW query LRs, i.e., Lq =FΘ(q)∈RC×HW . Then we calculate the relation matrix of these LRs as below:

where i ∈ {1, . . . ,HW}, j ∈ {1, . . . ,NKHW} and g(·, ·) is a similarity metric, which is implemented as cosine similarity in this paper. In contrast to previous methods that build image-level [Sung et al., 2018] or class-level [Li et al., 2019b] relationship, we aim to build a task-level relationship while maintaining discriminative relations at the same time.

Further more, we apply a transformation layer FΨ (i.e., the 1 × 1 conv layer in Figure 2) on the original LRs, and then learn another relation matrixMΨ for subsequent operations:

(2) where i ∈ {1, . . . ,HW}, j ∈ {1, . . . ,NKHW}. Each row in this matrix represents the adaptive subspace relationship of each position in the query image to all positions of all images in the support set. Moreover, we eliminate the noises (i.e., the trivial relations) in the relation matrixMΨ by a threshold Vc, and then produce an episodic attention map MA as below:

As Eq. (3) shown, the common patches shared by multiple classes among the entire task will "dilute" the attention, and thus they will enjoy relatively small attention values. Meanwhile, we find that although the influence of each noise (i.e., each trivial relation) is slight, it still greatly affects the distribution of the episodic attention due to the large number. For this reason, we apply Eq. (4) to construct a sparse episodic attention. In fact, this sparse episodic attention is more similar to a selection process or a hard attention rather than a soft attention. Next, we perform an element-wise multiplication between MA and MR to obtain a weighted relation matrix MA ⊙MR, and then collect the weighted relation between query q and the n-th class to obtain the score for n-th class:

where Vs is a temperature for the following cross-entropy loss, and Z n k indicates the k-th relation of KHW relations belong to the n-th class in the NKHW relations of the entire support set S. Finally, we can obtain the classification probability P q of query q by a softmax function. Note that based on the above process, we can develop the Task-aware Local Representations Network (TL-Net), which can be easily implemented by two matrix multiplications, an element-wise multiplication as well as some convolution operations.

3.3 Adaptive Threshold for Episodic Attention

In the sections above, we introduce the TL-Net, where a fixed threshold Vc (i.e., a global scalar) is used to select the most informative relationships. However, such kind of selection is sensitive to the value of Vc and not flexible for different query patches, which is shown in Figure 3. To handle this problem, we propose a novel adaptive episodic attention module FA, which can learn different thresholds for different patches.

Figure 2 shows the framework of our adaptive episodic attention module. Different from the method mentioned in Eq. (4), we use a Multi-Layer Perceptron (MLP) FΓ to adaptively predict the threshold for each LR of the query image. Specifically, FΓ takes the query LRs as input and outputs a threshold V ∗ c :

where σ is a sigmoid function. Beyond that, to narrow the search space for V ∗ c , we change the output range of sigmoid function σ. However, the step function used in Eq. (4) is indifferentiable. So we approximate it using a variant I ∗ (·) of sigmoid function with a hyperparameter k:

where V ∗ c is the corresponding threshold value for x, and x denotes one of the values in MA. Theoretically, when k is large enough, the I ∗ (·) can be considered as I(·). Moreover, we call the extended TL-Net with learnable thresholds as ATL-Net, Adaptive Task-aware Local Representations Network, to show its additional adaptive ability. The training process of the proposed ATL-Net is shown in Algorithm 1.

4 Experiments

4.1 Datasets

miniImageNet [Vinyals et al., 2016] is a subset of ImageNet [Deng et al., 2009], which consists of 100 classes and 600 images per class. Following the commonly used strategy, we divide the dataset into training (auxiliary)/validation/test set with a percentage of 64/16/20 respectively.

We also evaluate our method on three fine-grained image classification datasets. Stanford Dogs [Khosla et al., 2011] contains 120 categories with a total number of 20, 580 images. Stanford Cars [Krause et al., 2013] contains 196 classes of cars and 16, 185 images. CUB-200 [Welinder et al., 2010] contains 200 bird species with a total number of 6, 033 images. For fair comparisons, we use the data splits of [Li et al., 2019b; Li et al., 2019c; Huang et al., 2019], as Table 1 shows.

Algorithm 1 Training of ATL-Net

Input: Episodic task T = {AS,AQ} 1: while no converge do 2: LS ←FΘ(AS) 3: LQ ←FΘ(AQ) 4: for Lq in LQ do 5: Get relation matrixMR by Eq. (1) 6: Calculate adaptive threshold V ∗ c for Lq by Eq. (6) 7: Construct adaptive episodic attention MA by Eq. (2), Eq. (3) and Eq. (7) 8: Calculate probability P q for Lq by Eq. (5) 9: end for 10: L←- ∑ Y log(P) 11: Mini-batch Adam to minimize L, update Θ, Ψ and Γ 12: end while

4.2 Implementation Details

Network architecture. We follow the basic feature extraction network which is used in previous works [Li et al., 2019b; Li et al., 2019c]. The feature extraction network FΘ consists of 4 convolution blocks, each of which contains a convolutional layer, batch normalization and LeakyReLU activation. The transformation layer FΨ consists a 1 × 1 convolutional layer followed by batch normalization and LeakyReLU activation. The MLP module FΓ is implemented by two fully connected layers. In fact that only a few parameters are introduced by FΨ and FΓ, which will be discussed in Section 5.

Training and testing detail. We implement our experiments using PyTorch [Paszke et al., 2019]. All the images are resized to 84 × 84. During the training stage, we randomly construct 250, 000 episodes from the training (auxiliary) set for the miniImagenet dataset and the Stanford Car dataset, and 150, 000 for the other two datasets to avoid overfitting. In each episode, we collect 15 query images per class. For example, under 5-way 1-shot setting, we have 5 support images and 75 query images in each task. We use Adam [Kingma and Ba, 2015] optimizer with a cross-entropy loss to train the network. The initial learning rate is set to 0.001. During the test, we evaluate the proposed ATL-Net on 600 randomly sampled tasks. The mean accuracy, as well as the 95% confidence interval will be reported after being repeated five times. Note that the whole model is trained from scratch in an end-to-end manner without any data augmentation and weight decay, neither do fine-tune in the test stage 1 .

4.3 Baselines

To evaluate the proposed ATL-Net on the miniImagenet, we make comparisons with eleven state-of-the-art models, including Matching Net [Vinyals et al., 2016], MAML [Finn et al., 2017], Prototypical Net [Snell et al., 2017], GNN [Satorras and Estrach, 2018], Relation Net [Sung et al., 2018], MetaGAN [Zhang et al., 2018], MM-Net [Cai et al., 2018], MEPS [Chu et al., 2019], CovaMNet [Li et al., 2019c], DN4 [Li et al., 2019b] and GCR [Li et al., 2019a].

1 The source code can be available from https://github.com/ LegenDong/ATL-Net

For fine-grained image classification datasets, we compare our method with six few-shot methods, Matching Net [Vinyals et al., 2016], Prototypical Net [Snell et al., 2017], GNN [Satorras and Estrach, 2018], DN4 [Li et al., 2019b] and CovaMNet [Li et al., 2019c], and the fine-grained methods PABN+cpt/LRPABNcpt [Huang et al., 2019].

4.4 Comparisons with the SOTAMethods

We make comparisons with several state-of-the-art methods under 5-way 1-shot and 5-way 5-shot settings.

Results on miniImagenet. The results on miniImagenet are summarized in Table 2. It can be seen that our method significantly outperforms other methods under both settings. We achieve 54.30% under the 5-way 1-shot setting, with an improvement of 0.93% from the second best [Cai et al., 2018]. Moreover, compared with [Cai et al., 2018], the proposed ATL-Net introduces simpler additional structures (i.e., FΨ and FΓ) than the complex memory-addressing architectures. Similarly, our ALT-Net also gets higher performance, 0.88% improvement than previous methods [Li et al., 2019a] that uses data augmentation, data hallucination [Wang et al., 2018] and pretrains the feature extractor on the whole training set. Note that the proposed ATL-Net achieves an improvement of 3.06%/2.20% under 5-way 1-shot/5-shot settings than the most relevant work [Li et al., 2019b], which exploits the relation at the class-level by k-NN selection.

Such a great improvement further proves the superiority of our method that select the distinct patches, which are only shared between a certain class and query images.

Results on fine-grained datasets. The results on the three fine-grained datasets are summarized in Table 3. Due to the results for [Huang et al., 2019] on the CUB-200 [Welinder et al., 2010] dataset is not provided, we leave them blank. It can be observed that our method achieves the best performance compared with both general and fine-grained-specific fewshot learning methods. Compared with the general few-shot learning methods, our method is 5.39%, 8.11% and 8.49% better than the second best under the 5-way 1-shot setting. The results compared with the fine-grained few-shot learning methods are similar, we obtain 7.67% improvements at least. The reason for these great improvements is that ATLNet can naturally tackle the major challenge of identifying and weighting the importance of the key parts [Sun et al., 2018]. The proposed method will not be fooled by the similar global geometry and appearances, and thus pay more attention to their subtle differences behind the key parts.

4.5 Ablation Study

To further verify the effectiveness of the proposed ATL-Net, we conduct ablation studies on miniImagenet, the results are reported in Table 4. We remove FΨ and FΓ from the network respectively to confirm that each part of the model is indispensable, and them both as the baseline for comparison. It can be observed that the main improvement comes from the adaptive threshold module FΓ, without it about 1.06% and 1.35% performance loss occurring. Intuitively, FΨ can somewhat alleviate the effects of an unsuitable threshold for a certain relation, by measuring in an adaptive subspace without harming the original features. In contrast, FΓ tries to directly assign an adaptive threshold, making it more effective than FΨ. However, FΓ still cannot handle all changeable cases, so both FΓ and FΨ are indispensable. Moreover, even for the TL-Net (i.e., ATL-Net without FΓ), we also obtain competitive results with only 0.13% and 0.47% lower than [Cai et al., 2018] and [Li et al., 2019a] under 5-way 1-shot and 5-shot settings, but much better than the other methods.

5 Discussion

5.1 Value-based v.s. Rank-based Selection

To verify the superiority of value-based selection than rankbased one [Li et al., 2019b], we replace the adaptive attention module by a k-NN selection, the peak and surrounding results under 5-way 5-shot setting are reported in the Figure 3. The results of the value-based selection are generally better than the results of the rank-based selection. Meanwhile, we observe that it's difficult for the rank-based method to select enough corresponding LRs without too much noise at the task-level. For example, the rank-based methods cannot select enough corresponding LRs for k = 3 or 8 , while the corresponding LRs contain numerous noise when k = 32 or 64.

5.2 Influence of the Threshold Vc

The results of the threshold Vc are reported in Figure 3, which shows that the choice of the threshold Vc has a mild impact on performance, and we observe that the performance degradation due to information loss is more severe than the noise introduced. To verify the importance of episodic attention's sparsity, we replace the hard attention by a soft attention. The result shows that the trivial relations, even with relatively small attention values, still greatly affect the distribution of episodic attention. Beyond that, the proposed adaptive episodic attention module FA outperforms 1.35% than the best result with a manual selection for Vc. Such a great improvement shows that the global manually set Vc can only achieve a suboptimal solution, but the adaptive threshold V ∗ c is more robust in complex situations.

5.3 Number of Trainable Parameters

We also compare the number of trainable parameters to verify the efficiency of the proposed ATL-Net, as the Table 5 shows. Since no other trainable parameters are introduced except for the embedding module FΘ, [Snell et al., 2017; Li et al., 2019b] become the most light-weight models. [Satorras and Estrach, 2018] adopts a larger embedding module (i.e., the filter number is 256), which draws a great contribution from the number of parameters. [Sung et al., 2018; Li et al., 2019a] adopt additional architectures to boost the result, which also introduce a huge number of trainable parameters. However, the proposed ATL-Net only introduces a small number of the trainable parameters, while achieves a better result than the methods above.

6 Conclusions

In this paper, we propose an Adaptive Task-aware Local Representation Network (ATL-Net) for few-shot learning, aiming to learn more discriminative local representations by taking a view of the entire task. Specifically, an adaptive episodic attention mechanism is designed to adaptively select the key semantic patches for a special task, without distracting attention by the common parts shared by most classes. Extensive experimental results on the benchmarks verify the effectiveness and superiority of the proposed ATL-Net.

Acknowledgements

This work is supported by National Key R&D Program of China (2018YFB1402600), NSFC (61806092) and Jiangsu Natural Science Foundation (No. BK20180326).