Few-Shot Learning with Part Discovery and Augmentation from Unlabeled Images

1 Introduction

Recently, great progress in the computer vision community has been achieved with deep learning, which often needs numerous training data. Unfortunately, there are many practical applications where collecting data is very difficult. To learn a novel concept with only few examples, few-shot learning has been recently proposed and gains extensive attention.[Doersch et al., 2020; Afrasiyabi et al., 2020]

A possible solution to few-shot learning is meta-learning [Finn et al., 2017; Vinyals et al., 2016; Ravi and Larochelle, 2017; Snell et al., 2017; Peng et al., 2019]. It first extracts shared prior knowledge from many similar tasks. After that, this knowledge will be adapted to the target few-shot learning task to restrict the hypothesis space, which makes it possible to learn a novel concept with few examples. Equipped with neural networks, the prior knowledge can be parameterized as an embedding function, or a set of initial parameters of a network, which often needs an extra labeled base dataset for training. Another line of work is based on transfer learning [Tian et al., 2020; Dhillon et al., 2020], which extracts prior knowledge as a pre-trained feature extractor. Typically, a standard cross entropy loss is employed to pre-train the feature extractor, which also needs a labeled base dataset. However, collecting a large labeled dataset is time-consuming and laborious. Besides, these labels are sadly discarded when performing a target few-shot learning task, because they belong to different class spaces. Inspired by recent progress of unsupervised learning, a question is naturally asked: can we can learn prior knowledge only from unlabeled images? If yes, it will be a promising approach for the scenario where many unlabeled images are available but the target task is data-scarce.

∗Contact Author

Some remarkable works have made an effort for this purpose, e.g., unsupervised meta-learning [Hsu et al., 2019; Khodadadeh et al., 2019]. However, these unsupervised methods are hindered by learning effective class-related representations from images, compared to the supervised counterparts. This is because much unrelated or interfering information, e.g., background clutters, may impose adverse impacts on representation learning under label-free unsupervised setting. Selecting discriminative image regions or target parts is an effective way to reduce this interference during representation learning, which has a consistent motivation with traditional part-based models [Felzenszwalb et al., 2009].

In this paper, we propose a part-based self-supervised learning model, namely Part Discovery Network (PDN), to learn more effective representations from unlabeled images. The key point of this model is mining discriminative target parts from images. Due to the lack of part labels, multiple image regions are first extracted via random crop, which inevitably contain interfering backgrounds or less informative regions. To eliminate these regions, we choose the image region as the most discriminative target part, which keeps the largest average distance to other images (negative samples). The rationale of this selection is that the discriminative part should be able to distinguish the original image from others [Singh et al., 2012], so its average distance to other images should be large enough. With the selected discriminative part, we maximize its similarity to the original image in a similar way to [He et al., 2020].

Another challenge in the few-shot scenario is that the target classifier is easy to overfit due to the scarcity of support training samples. An effective way to prevent overfitting is data augmentation, which has been explored in the few-shot learning literature [Schwartz et al., 2018; Wang et al., 2018]. However, most of these methods assume that the variance of seen classes can be directly transferred to unseen classes, which is too strict in most situations. In this paper, we resort to retrieve extra images from a base dataset and extract their part features to augment support set, based on the fact that similar objects generally share some common parts. The core of our method is selecting these part features which match well with image features in the support set. Specifically, we propose a novel Class-Competitive Attention Map (C2AM) to guide the relevant part selection from the retrieved images, and then refine target classifier with these selected parts. Our method is also called Part Augmentation Network (PAN), and Figure 1 illustrates the main idea of PAN.

Our Part Discovery and Augmentation Network (PDANet) consisting of both PDN and PAN largely outperforms the stat-of-the-art unsupervised few-shot learning methods, and achieves the comparable results with most of the supervised methods. The contributions of this work can be summarized as:

• We propose a novel self-supervised Part Discovery Network, which can learn more effective representations from unlabeled images for few-shot learning.
• We propose a Part Augmentation Network to augment few support examples with relevant part features, which mitigates overfitting and leads to more accurate classification boundaries.
• Our method outperforms previous unsupervised methods on two standard few-shot learning benchmarks. Remarkably, our unsupervised PDA-Net is also comparable with supervised methods.

2 RelatedWork

Few-Shot Learning. Few-shot learning aims at learning a novel concept from few examples. A possible solution is meta-learning, which extracts prior knowledge from many similar tasks (episodes). For example, MAML [Finn et al., 2017] learns the optimal initial parameters of a network, which can be quickly adapted to a new task via gradient descent. Matching Networks [Vinyals et al., 2016] classifies an instance based on its nearest labeled neighbor in the learned embedding space. Different from meta-learning, [Dhillon et al., 2020] demonstrate that a strong transfer learning baseline can achieve competitive performance. In order to train models, all of the above methods need a large labeled base dataset. Unsupervised meta-learning methods loose this constraint by constructing training episodes from unlabeled images. [Hsu et al., 2019] propose to acquire pseudo labels by clustering in an unsupervised feature space. [Khodadadeh et al., 2019] use random sampling and augmentation to create synthetic episodes. Our method aims at both learning more effective representation from unlabeled images and increasing support set by data augmentation for unsupervised few-shot learning.

Self-Supervised Learning. Self-supervised learning aims at leveraging unlabeled images to learn good representations for down-stream tasks. Previous work mainly focuses on mining supervision signals from unlabeled data [Gidaris et al., 2018; Doersch et al., 2015; Zhang et al., 2016]. Recently, contrastive learning shows superior performance improvement, which maximizes the similarity of two different views of the same image on global level [Chen et al., 2020] or local level [Ouali et al., 2020], or enforces consistency of cluster assignments between two views [Caron et al., 2020]. In this paper, we explicitly mine discriminative parts for contrastive learning, which learns more effective representations from unlabeled images.

Data Augmentation for Few-Shot Learning. A straightforward way to deal with data deficiency is to synthesize more data. [Schwartz et al., 2018] propose to extract intra-class variance (deltas) from base classes and use them to synthesize samples for novel classes. [Wang et al., 2018] encode the intra-class variance in a hallucinator, which can synthesize new features taking as input a reference feature and noise vectors. To leverage base class samples, [Chen et al., 2019] propose to generate a deformed image by fusing a probe image and a gallery image. Our method augments support set by retrieving extra images from a base dataset and extracting matched part features with the support samples.

3 Notation

In this section, we briefly illustrate the formulation of the fewshot image classification problem. Given a labeled support set S = {(xi, yi)}Ns i=1 where xi ∈ I is an image and yi ∈ Cnovel is its label, we are supposed to predict the labels of a query set Q = {(xi, yi)}Nq i=1 , which also belongs to Cnovel. The number of classes |Cnovel| is called way and the number of samples in each class is called shot. For few-shot learning, the shot is very small, like 1-shot or 5-shot. Due to the scarcity of support samples, it is very hard to train a classification model from scratch. Therefore, we are given an extra large base dataset Dbase to mine prior knowledge. Here, Dbase is from base classes Cbase, and Cbase ∩ Cnovel = φ. Previous works usually need base labels to construct training episodes or pre-train the classification model. In this paper, we only use unlabeled images in Dbase , i.e., Dbase = {xi}Nb i=1 .

4 Method

Our method follows a transfer learning protocol which includes two stages, namely the representation learning stage and the few-shot learning stage. On the first stage, we aim at learning a feature extractor f from Dbase . On the second stage, the learned feature extractor f is transferred to the target few-shot learning task, followed by training a linear classifier using support set S. The final classification results of query set Q are predicted by the learned classifier. From the above description we can see that the key is to learn a good feature extractor from base dataset Dbase and meanwhile obtain a robust classifier with limited data in support set S. To this end, we propose Part Discovery and Augmentation Network (PDA-Net), which consists of a Part Discovery Network (PDN) to learn effective representations, and a Part Augmentation Network (PAN) to augment few support examples with relevant part features. The whole framework is shown in Figure 2.

4.1 Part Discovery Network

Our Part Discovery Network (PDN) is a part-based selfsupervised learning model. We will introduce how to select parts and training details as follows.

Extracting Multiple Parts

Without part labels, we generate parts by randomly cropping a given image x into n patches {xpi }ni=1 . Meanwhile, a larger crop xg is obtained to serve as global context. Random transformations are further applied to increase data diversity. To get latent part representations {hpi }ni=1 and global representation hg , a convolution neural network f is applied followed by global average pooling and a MLP projection head.

Selecting Discriminative Part

Since we generate multiple parts with random crop, there are inevitably some crops that belong to background. Directly matching these crops to the global view will create bias towards backgrounds and hurt the generalization of the learned representations. To solve this problem, we develop an effective strategy to select the most discriminative part. Given a set of negative samples D- = {hgi }N - i=1 that are exclusive with the input image x, we define a sample-set distance metric d(hpi ,D-) in the feature space, which indicates the distance between a part and all negative samples. We select the part with the maximum distance as the most discriminative one, therefore excluding background crops and less informative ones. The strategy is formulated as

where hp is the selected part. The rationale of this strategy is that the discriminative part should be able to distinguish the original image from others, so its distance to other images should be very large.

The distance between hpi and D- can be defined in many forms. One could use the minimum distance between hpi and all samples in D-. However, some similar images may exist in D-, so the minimum distance is severely affected by these similar images and can not reflect the true distance to most negative samples. Here we choose the mean distance to calculate d(hpi ,D-), which considers more on the samples of other classes. d(hpi ,D-) is calculated as:

where |D-| is the number of negative samples in this set, and s is the cosine similarity.

Training

With selected discriminative parts, we train the PDN with a contrastive loss, which is formulated as

where τ denotes a temperature hyper-parameter.

To get a large negative set, we follow [He et al., 2020] to organize D- as a queue and use a momentum encoder to get consistent negative representations. The momentum encoder is an exponential moving average version of the feature extractor and MLP head. Its parameter θm is updated as

where m is a momentum hyper-parameter, and θt denotes the parameters of the feature extractor and MLP head at training step t.

4.2 Part Augmentation Network

Our Part Augmentation Network (PAN) aims to augment support set with relevant part features from base dataset. To this end, we first retrieve extra images from base dataset, then create a Class-Competitive Attention Map (C2AM) to guide the relevant part selection from the retrieved images, and finally refine target classifier with these selected parts.

Retrieving Extra Images

Since the size of Dbase is very large, we propose a simple but effective strategy to select a small number of very similar images from Dbase , which are more likely to contain relevant parts. Specifically, we first train a linear classifier p(y|z; W , b) on support set S, where z denotes a feature vector, W and b are weight matrix and bias, respectively. Then, we employ this classifier to classify each image in Dbase as:

where GAP is a global average pooling operator and M denotes the feature map extracted by the learned feature extractor f in Section 4.1

Among the images of class k in Dbase , we keep the Na images with the highest classification probability as the retrieval results. The feature maps of these Na images are denoted as Ak = {Mk i }Na i=1 .

Class-Competitive Attention Map

To further extract relevant parts from retrieved feature maps Ak, we propose a novel CAM-based [Zhou et al., 2016] attention mechanism, Class-Competitive Attention Map (C2AM), illustrated in Figure 3. Given a feature map M k ∈ Ak, we obtain a class attention map Sk that indicates its relevance to class k at each spatial location:

where Sk (i, j) and M k (i, j) are the classification score for class k and the feature vector at location (i, j), respectively. W k , bk are the classifier weight vector and bias for class k, which have been learned in p(y|z;W, b).

Although the class attention map Sk is very useful to locate class-specific image regions or object parts, it still contains some parts that have high classification scores for all classes. These parts provide less information for classification, and should be further inhibited. We perform softmax over the classification score vector S(i, j) at each spatial location, which provides a class competitive mechanism to highlight the parts which only have higher score for class k. The classcompetitive attention map αk for class k is as follows:

With this revised attention map αk , we can extract more relevant part features zk to augment the class k, which is calculated by the weighted sum of feature map M k :

The retrieved feature set for class k is now updated as Ak = {zk i }Na i=1 . We denote A = ∪kAk as the set of augmented part features for all classes.

Refining Target Classifier

With augmented part features, we can now refine the initial classifier with both support set S and augmented set A to get a more robust classifier. Since augmented features are not from the exactly same classes as support samples, we adopt the label smoothing technique [Szegedy et al., 2016] to prevent overfitting on the augmented features. Specifically, we convert an one-hot label y into a smoothed probability distribution py:

where py(k) is the probability of the class k, and ϵ is a hyperparameter in range (0, 1). Finally, the total loss for refining the classifier is:

where the first term is cross entropy loss for samples in support set S, and the second term is K-L divergence between predicted probability distribution p(·|z) and smoothed ground-truth distribution py for augmented features.

5 Experiments

5.1 Datasets

miniImageNet. miniImageNet is a standard benchmark for few-shot learning proposed by [Vinyals et al., 2016]. It is a subset of the ImageNet [Russakovsky et al., 2015] and contains 100 classes and 600 examples for each class. We follow the protocol in [Ravi and Larochelle, 2017] to use 64 classes for training, 16 classes for validation and 20 classes for test.

tieredImageNet. tieredImageNet [Ren et al., 2018] is a larger subset of ImageNet and contains 608 classes and 1000 images in each class. Theses classes are grouped into 34 higher categories, where 20 categories (351 classes) for training, 6 categories (97 classes) for validation and 8 categories (160 classes) for test. The large semantic difference between categories makes it more challenging for few-shot learning.

5.2 Implementation Details

For PDN, we transform input images with random crop, horizontal flip, color jitter and Gaussian blur. The crop scales for part view and global view are in range of (0.05, 0.14) and (0.14, 1), respectively. The number of cropped parts is set as n = 6 by cross validation. The size of D- is 1024 and 10240 for miniImageNet and tieredImageNet, respectively. We use standard ResNet-50 as backbone and resize images to 224 × 224. We set hyper-parameter m = 0.999 and τ = 0.2. We adopt SGD optimizer with cosine learning rate decay. The learning rate is 0.015 for miniImageNet and 0.03 for tieredImageNet.

For PAN, we retrieve Na = 1024 extra images for each class. The label smoothing hyper-parameter ϵ is 0.2 for 1shot and 0.7 for 5-shot. The loss weight λ is set as 1. The classifier is trained with Adam and the learning rate is 0.001. Following the similar setting to [Vinyals et al., 2016], We evaluate our method on 600 test episodes.

5.3 Comparison with PriorWork

In Table 1, we compare our method with both supervised and unsupervised few-shot learning methods. Overall, our method achieves the best performance under unsupervised setting, and is comparable with state-of-the-art supervised methods.

Our PDA-Net significantly outperforms unsupervised meta-learning [Hsu et al., 2019] by 23.91% and 29.14% on miniImageNet in 1-shot and 5-shot, respectively. Compared with the most related method which also employs MoCo for representation learning [Tian et al., 2020], we achieve 9.65% and 10.07% improvement in 1-shot and 5-shot on miniImageNet. On the larger tieredImageNet, our method largely outperforms a recent GAN-based method, GdBT2 [Khoi and Sinisa, 2020], by 21.15% and 16.50% in 1-shot and 5-shot, respectively. The significant improvements over the compared unsupervised methods verify the effectiveness of our PDA-Net.

Furthermore, our method outperforms two supervised meta-learning methods, LEO [Rusu et al., 2019] and MetaOptNet [Lee et al., 2019], on both datasets in both 1shot and 5-shot. Compared with the supervised transfer learning method, Distill [Tian et al., 2020], the performance of our PDA-Net only drops 0.98% on miniImageNet in 1-shot, which is acceptable considering the expensive label budget. Alignment [Afrasiyabi et al., 2020] uses base labels to detect the most related class, which achieves better performance in 1-shot, but is comparable with ours in 5-shot.

5.4 Further Analysis

Ablation Study. We evaluate the effectiveness of each key component of PDA-Net on miniImageNet in 1-shot, and show the results in Table 2. The baseline is constructed by ablating part discovery and part augmentation, thus only taking as input two global views for contrastive learning. PDN w/o Select means that we use all the 6 parts for contrastive learning, while PAN w/o C2AM means that we use class activation map instead [Zhou et al., 2016]. Compared with the baseline, PDN w/o Select obtains 2.20% performance gain, which verifies the advantage of part-based representation learning. PDN can get further improvement via selecting the most discriminative part. Better results are achieved by adding PAN to PDN or PDN w/o Select, which illustrates the effectiveness of our part augmentation strategy. Compared with CAM in [Zhou et al., 2016], our C2AM can obtain better performance, demonstrating the effectiveness of our proposed class competitive mechanism.

Number of Crops. In PDN, we use random crop to get n parts. Here we evaluate the impact of number of cropped parts on representation learning. As shown in Table 3, the performance rapidly increases with the growing number of cropped parts, because it is more likely to contain discriminative parts with more cropped parts. The performance gets saturated after reaching 6, which indicates that our selection strategy is an effective way to extract discriminative parts.

Number of Augmented Features. In PAN, we retrieve Na extra images to augment each novel class. Here we experiment with different Na and show the results in Figure 4. It can be seen that the benefit of augmented features is more significant for 1-shot than 5-shot. For 1-shot, the performance grows rapidly as Na increases and finally gets saturated after Na = 1024, which is because relevant features in base classes are limited. With a larger base dataset, we can infer that our method can get better results.

5.5 Visualization

To explore the discriminative part selection, we visualize the cropped parts and their corresponding global view in Figure 5. It should be noted that these images are all randomly transformed, so they may have different appearances even from the same input image. We sort the cropped parts based on the distances to negative samples and select the parts with the largest distances within red boxes. For example, the main body of a bird and the head of a dog are selected in the first and fourth rows, respectively. We can see that as the distances decrease, the parts are more likely to belong to background and contain less information about target object.

To illustrate the rationality of part augmentation strategy, we visualize the retrieved unlabeled images and their classcompetitive attention maps in Figure 6. We can see that the retrieved images are usually similar to support images. More interestingly, C2AM can locate class-specific image regions which are very relevant to the objects in support set. For example, the antennae and legs of the ladybug are highlighted in the retrieved image, which are very similar to those of the ant in support image. In contrast, its shell attracts less attention due to the distinctive appearance from the ant.

6 Conclusion

In this paper, we present a Part Discovery Network, which can extract effective prior knowledge for few-shot learning from a flat collection of unlabeled images. Furthermore, a Part Augmentation Network is proposed to augment support examples with relevant parts, which can mitigate overfitting and lead to better classification boundaries. The experimental results demonstrate that our method significantly outperforms previous unsupervised meta-learning methods and achieves comparable accuracy with state-of-the-art supervised methods.

Acknowledgements

This work is supported by National Natural Science Foundation of China (61976214, 61633021, 61721004 , 61836008), Beijing Municipal Natural Science Foundation (4214075), and Science and Technology Project of SGCC Research on feature recognition and prediction of typical ice and wind disaster for transmission lines based on small sample machine learning method.