Asymmetric Distribution Measure for Few-shot Learning

1 Introduction

Few-shot learning (FSL) for image classification has gained considerable attention in recent years [Vinyals et al., 2016; Finn et al., 2017; Sung et al., 2018; Lee et al., 2019], which attempts to learn a classifier with good generalization capacity for new unseen classes with only a few samples. Because of the scarcity of data, it is almost impossible to directly train a conventional supervised model (e.g., a convolutional neural network) from scratch by only using the few available samples. Therefore, transfer learning shall be a natural way to learn transferable knowledge to boost the target few-shot classification. Along this way, a variety of methods have been proposed, which can be roughly divided into three categories: data-augmentation based methods [Antoniou et al., 2017; Schwartz et al., 2018; Xian et al., 2019], meta-learning based methods [Ravi and Larochelle, 2017; Jamal and Qi, 2019; Lee et al., 2019] and metric-based methods [Vinyals et al., 2016; Sung et al., 2018; Li et al., 2019b]. Metric-based FSL methods have achieved significant successes and attracted increasing attention due to their simplicity and effectiveness. In this work, we focus on this kind of methods.

∗Corresponding author

The basic idea of metric-based FSL methods is to learn a transferable deep embedding network by directly measuring the relations between query images and support classes. Thus, two key issues are involved in such a kind of methods, i.e., feature representations and relation measure. For feature representations, traditional methods such as ProtoNet [Snell et al., 2017] and RelationNet [Sung et al., 2018] generally adopt image-level global feature representations for both query images and support classes. However, due to the scarcity of samples in each class, the distribution of these image-level global features cannot be reliably estimated. Recently, CovaMNet [Li et al., 2019b] and DN4 [Li et al., 2019a] introduce deep local descriptors into FSL and attempt to utilize the distribution of local descriptors to represent each support class, which have been verified to be more effective than using the image-level global features.

On the relation measure, the existing methods including the above methods usually adopt an instance-level metric, where the query image is taken as one single instance (i.e., an imagelevel feature representation) or a set of instances (i.e., a set of local feature descriptors). For example, in ProtoNet, the Euclidean distance is chosen to calculate the distance between a query instance and the prototype (i.e., mean vector) of each support class. Also, CovaMNet proposes a covariance metric function to measure a local similarity between each local descriptor of a query image and a support class. Afterwards, it aggregates all the local similarities to obtain a global similarity as the relation between this query image and this class.

However, these existing methods have only considered the distributions of the support classes while neglecting the natural distribution of the local descriptors from a query image. Moreover, the instance-level metric they employ can only capture a kind of local relations (i.e., local similarities) between the query images and support classes. We argue that the distribution associated with a query image is equally important and a distribution-level measure shall be designed to capture the global-level relations between a query and a class. More importantly, we observe that the existing methods usually adopt a symmetric metric function (i.e., M(a, b) = M(b, a)) to calculate a symmetric relation between a query and a class. For instance, both the Euclidean distance used in ProtoNet and the cosine similarity adopted in CovaMNet and DN4 are symmetric functions. However, we highlight that there is naturally an asymmetric relation between a query image and a certain class. In particular, when each image is represented by a set of deep local descriptors, the distribution of the descriptors in one query image is only comparable to part of the distribution of the descriptors extracted from a support class. Therefore, we argue that an asymmetric measure is more suitable for the metric-based FSL to capture the asymmetric relations.

To this end, we develop a novel Asymmetric Distribution Measure (ADM) network for metric-based FSL. First, we represent each image as a set of deep local descriptors (instead of a single image-level global feature) and consider characterizing both a query image and a support class from the perspective of local descriptor based distributions (e.g., a Gaussian distribution with mean vector and covariance matrix). Second, we employ an asymmetric Kullback-Leibler (KL) divergence measure to align the distribution of a query with the distribution of a support class to capture the global distribution-level asymmetric relations. Third, to further improve the metric by taking the context of the task into consideration, we propose a task-aware Contrastive Measure Strategy (CMS), which can be used as a plug-in to any measure functions. Finally, inspired by the successful image-to-class measure (an asymmetric measure as a whole) introduced in DN4 [Li et al., 2019a] which mainly captures the asymmetric relations via individual local descriptor based cosine similarity measures, we combine the whole distribution based KL divergence measure with the image-to-class measure together to simultaneously capture the global and local relations.

The main contributions of this work are as follows:

• We propose a pure distribution based method for metricbased FSL and show that an asymmetric measure is more suitable for this kind of FSL methods.
• We simultaneously combine the global relations (i.e., the KL divergence measure) and the local relations (i.e., the image-to-class measure) together to measure the complete asymmetric distribution relations between a query and a class.
• We propose an adaptive fusion strategy to adaptively integrate the global and local relations.
• We design a task-aware contrastive measure strategy (CMS) as a plug-in to further enhance the adopted measure functions.

2 RelatedWork

We first briefly review metric-based FSL methods in the literature, and then introduce related work that inspires our work.

The first metric-based FSL method was proposed in [Koch et al., 2015], which adopts a Siamese neural network to learn transferable and discriminative feature representations. In [Vinyals et al., 2016], a Matching Net which directly compares a query image with a support class was presented, where a subsequently widely used episodic training mechanism was also proposed. After that, [Snell et al., 2017] proposed a ProtoNet, which represents a support class by a prototype, i.e., the mean vector of all sample in this class. Then a specific metric, i.e., Euclidean distance, was used to perform the final classification. Recently, based on ProtoNet, an infinite mixture prototypes (IMP) network was proposed [Allen et al., 2019], where each support class was represented by a set of adaptive prototypes. In addition, to avoid choosing a specific metric function, RelationNet [Sung et al., 2018] proposed to learn a metric through a deep convolutional neural network to measure the similarity between queries and support classes.

The above methods are all based on image-level feature representations. Due to the scarcity of samples in each class in FSL, the distribution of each class cannot be reliably estimated in a space of image-level features. Some recent work, such as CovaMNet [Li et al., 2019b] and DN4 [Li et al., 2019a] shows that the rich local features (i.e., deep local descriptors) can achieve better representations than the imagelevel features, because the local features can be regarded as a natural data augmentation operation. CovaMNet employs the second-order covariance matrix of the extracted deep local descriptors to represent each support class and designs a covariance-based metric to measure the similarities between a query image and a support class. Different from CovaMNet, DN4 argues that the pooling of local features into a compact image-level representation will lose considerable discriminative information. Therefore, DN4 proposes to directly use the raw local descriptor sets to represent both query images and support classes, and then employs a cosine-based image-toclass measure to perform the relation measure.

Inspired by CovaMNet and DN4, our ADM also takes the raw and rich deep local descriptors to represent an image. Compared with CovaMNet, the key difference is that CovaMNet only considers the distribution associated with a support class but neglect the distribution associated with a query image, while we consider both. Another important difference is that both CovaMNet and DN4 employ a cosine similarity function (i.e., a symmetric instance-level metric) to calculate a series of local relations between a query image and a certain class. In contrast, our ADM can capture the complementary global relations by using an extra distribution-level measure. In addition, we observe that the relation between a query image and a certain class is actually asymmetric, i.e., a query image is only commensurate with a sample in an image class when it is viewed as a set. Therefore, we argue that an asymmetric measure shall be considered for metric-based FSL to reflect this property.

3 Preliminary

Problem formulation. Under the few-shot setting, there are usually three sets of data, i.e., a support set S, a query set Q and an auxiliary set A. In particular, S and Q share the same label space, which are corresponding to the training and test sets respectively in the general classification task. If S contains C classes with K (e.g., 1 or 5) samples per class, we call this classification task C-way K-shot. However, S only has a few samples in each class, making it almost impossible to train a deep neural network effectively. Therefore, the auxiliary set A is generally introduced to learn transferable knowledge to tackle this problem. Also, A enjoys more classes and more samples per class than S, but has a disjoint label space from S.

Episodic training. To learn a classifier that can generalize well, an episodic training mechanism [Vinyals et al., 2016] is normally adopted in the training stage of the metric-based FSL methods. Specifically, in each episode, a new task simulating the target few-shot task is randomly constructed from A. Each simulated task consists of two subsets, AS and AQ, which are akin to S and Q, respectively. At each iteration, one episode (task) is adopted to train the current model. Basically, tens of thousands of episodes (tasks) will be randomly sampled to train this model. Once the training process is completed, we can predict the labels of Q using the trained model based on S.

4 Methodology

As illustrated in Figure 1, our ADM model mainly consists of three components: a feature embedding module, a joint asymmetric measure module, and a classifier module. The first module learns feature embeddings and produces rich deep local descriptors for an input image. Afterwards, the distributions of each query image and each support class can be represented at the level of deep local descriptors. The second module defines a joint asymmetric distribution measure between a query's distribution and a support class's distribution by considering both the asymmetric local and global relations. As for the last module, we adaptively fuse the local and global relations together by a jointly learned weight vector, and then adopt a non-parametric nearest neighbor classifier as the final classifier. These three modules are jointly trained from scratch in an end-to-end manner.

4.1 Feature Embedding with Local Descriptors

As have been shown by some recent work [Li et al., 2019b; Li et al., 2019a], local descriptor based feature representations are much richer than image-level features and can alleviate the scarcity issue of samples in FSL. Following the above work, we employ the rich and informative local descriptors to represent each image as well.

To this end, we design a feature embedding module fϕ(·), which can extract deep local descriptors for input images. Specifically, given an image X, fϕ(X) will be a c × h × w three-dimensional (3D) tensor, which can be seen as a set of c-dimensional local descriptors

where xi is the i-th local descriptor and n = h ×w is the total number of local descriptors for image X. These local descriptors can be seen as the local representations of the spatial local patches in this image. Basically, for each query image, we use the extracted n local descriptors to estimate its distribution in the space of Rc . As for each support class, all the local descriptors of all the images in this class will be used together to estimate its distribution in the space of Rc . Since the local descriptors can capture the local subtle information, they can benefit more for the final image recognition.

4.2 Our Asymmetric DistributionMeasure (ADM) Kullback-Leibler divergence based distribution measure.

Assuming that the distribution of local descriptors extracted from an image or a support class is a multivariate Gaussian, a query image's distribution can be denoted by Q = N(µQ, ΣQ), and a certain support class's distribution can be expressed by S = N (µS , ΣS ), where µ ∈ Rc and Σ ∈ Rc×c indicate the mean vector and covariance matrix of a specific distribution, respectively. Thus, Kullback-Leibler (KL) divergence [Duchi, 2007] between Q and S can be defined as:

where trace(·) is the trace operation of matrix, ln(·) denotes logarithm with the base of e, and det indicates the determinant of a square matrix. As seen, Eq.(2) takes both the mean and covariance into account to calculate the distance between two distributions.

Typically, since the KL divergence measure is asymmetric, DKL(Q‖S) mainly matches the distribution of Q to the one of S, which is essentially different from DKL(S‖Q). One important advantage of using Eq.(2) is that it can naturally capture an asymmetric relation between a query image and a support class, forcing the query images to be close to the corresponding true class when used in our network training.

To further show the advantage of using an asymmetric measure, we purposely introduce a symmetric distribution metric function, e.g., 2-Wasserstein distance [Olkin and Pukelsheim, 1982], whose formulation is defined as follows,

However, due to the square rooting of matrices, the calculation of the above distance function is time consuming and the optimization of this function is difficult. Therefore, in the literature [Berthelot et al., 2017; He et al., 2018], an approximation function is normally employed

where the first term calculates the squared Euclidean distance between two mean vectors and the second term is a squared Frobenius norm of the difference between two covariance matrices. The comparison and analysis between 2-Wasserstein distance and KL divergence will be detailed in Section 5.5.

Image-to-Class based distribution measure. The above KL divergence measure can capture the global distributionlevel relation between a query image and a support class. Nevertheless, the local relations are not taken into consideration yet. According to a deep analysis of DN4 [Li et al., 2019a], we observe that there may be two implicit reasons for the success of DN4. One reason is that the local descriptor based measure (i.e., local relations) it used enjoys a stronger generalization ability than the image-level feature based measure. The other key reason is that the image-to-class measure used in DN4 is asymmetric on the whole, which aligns well with our argument of the necessity of the asymmetric measure. Therefore, such an asymmetric image-to-class measure is also introduced into our model to capture the local-level relations between a query and a support class. However, the difference in our work lies that the indispensable global relation is also measured by an asymmetric distribution-level measure (i.e., KL divergence).

To be specific, given a query image Q and a support class S, which will be represented as fϕ(Q) = [x1, . . . , xn] ∈ Rc×n and fϕ(S) = [fϕ(X1), . . . , fϕ(XK)] ∈ Rc×nK , respectively, where K is the number of shots in S. Thus, the image-to-class (I2C) similarity measure can be formulated as

where Topk(·) means selecting the k largest elements in each row of the correlation matrix between Q and S, i.e., fϕ(Q)⊤·fϕ(S) ‖fϕ(Q)‖F ·‖fϕ(S)‖F . Typically, k is set as 1 in our work.

Classification with an adaptive fusion strategy. Since two types of relations have been calculated, i.e., global-level relations calculated by the KL divergence measure and locallevel relations produced by the I2C measure, a fusion strategy shall be designed to integrate these two parts. To tackle this issue, we adopt a learnable 2-dimensional weight vector w = [w1, w2] to implement this fusion. It is worth noting that because the KL divergence indicates dissimilarity rather than similarity, we use the negative of this divergence to obtain a similarity. Specifically, the final fusion similarity between a query Q and a class S can be defined as follows

As seen in Figure 1, for a 5-way 1-shot task and a specific query Q, the output of the I2C branch or KL branch is a 5-dimensional similarity vector. Next, we concatenate these two vectors together to get a 10-dimensional vector. After that, we apply a 1D convolution layer with the kernel size of 1×1 along with a dilation value of 5. In this way, we can obtain a weighted 5-dimensional similarity vector by learning a 2-dimensional weights w. Additionally, a Batch Normalization layer is also added before the 1D convolution layer to balance the scale of the two parts of similarities. Finally, a non-parametric nearest neighbor classifier is performed to obtain the final classification results. Also, a cross-entropy loss is used to learn the entire network.

4.3 Our Contrastive Measure Strategy (CMS)

To make the distribution measure more discriminative, we further propose an alternative task-aware Contrastive Measure Strategy (CMS) by introducing additional contrastive information. Specifically, for a support set S = {S1, · · · , SC}, where C is the number of classes in S, we construct a distribution-level triplet 〈Q, Si, S′ i 〉. In this triplet, Q denotes a query's distribution, Si is the distribution of one class we want to match Qwith, and S′ i indicates the entire distribution of the remaining classes Sj|Cj=1 (j ̸= i). In this way, we can define the contrastive KL divergence measure as follows

The advantage of using the above contrastive measure function over merely using DKL(Q‖Si) in Eq.(2) is that the context of the entire support classes is taken into consideration. In this way, we can take a whole view of the entire task when measuring the relation between Q and each individual class Si, making the measure function more discriminative. This will be experimentally demonstrated shortly.

5 Experimental Result

5.1 Datasets

All experiments are conducted on both miniImageNet [Vinyals et al., 2016] and tieredImageNet [Ren et al., 2018]. miniImageNet is widely used in FSL, which is a small subset of ImageNet [Deng et al., 2009]. It contains 100 classes with 600 images in each class. We use the same splits as in [Ravi and Larochelle, 2017], which takes 64, 16 and 20 classes for training, validation and test, respectively.

tieredImageNet is also a mini-version of ImageNet. Different fromminiImageNet, tieredImageNet has a larger number of classes (608 classes) and more images for each class (1281 images per class). On this dataset, we strictly follow the splits used in [Ren et al., 2018], which takes 351, 97 and 160 classes for training, validation and test, respectively.

For both miniImageNet and tieredImageNet, the resolution of all the images is resized to 84×84.

5.2 Network Architecture

It can be easily verified that adopting a deeper network for embedding or using pre-trained weights will provide higher accuracy. Following the previous works [Snell et al., 2017; Sung et al., 2018; Li et al., 2019b; Li et al., 2019a], we adopt the same embedding network with four convolutional blocks, i.e., Conv-64F, to make a fair comparison with other methods. Specifically, each of the first two blocks contains a convolutional layer (with 64 filters of size 3 × 3), a batchnormalization layer, a Leaky ReLU layer and a max pooling layer. The last two blocks adopt the same architecture but without pooling layers. The reason for only using two pooling layers is that we need richer local descriptors to represent the distributions of both queries and classes. For example, in a 5-way 1-shot setting, when the size of the input image is 84 × 84, we can only obtain 25 local descriptors for each image (class) by adopting four pooling layers. It is clearly insufficient to represent a distribution with a feature dimensionality of 64. In contrast, using the adopted network architecture with two pooling layers, we obtain 441 local descriptors for each image (class).

5.3 Implementation Details

Both 5-way 1-shot and 5-way 5-shot classification tasks are conducted to evaluate our methods. We use 15 query images per class in each single task (75 query images in total) in both training and test stages. In particular, we employ the episodic training mechanism [Vinyals et al., 2016] to train our models from scratch without pre-training. In the training stage, we use the Adam algorithm [Kingma and Ba, 2014] to train all the models for 40 epoches. In each epoch, we randomly construct 10000 episodes (tasks). Also, the initial learning rate is set as 1 × 10-3 and multiplied by 0.5 every 10 epoches. During test, 1000 tasks are randomly constructed to calculate the final results, and this process is repeated five times. The top-1 mean accuracy is taken as the evaluation criterion. At the same time, the 95% confidence intervals are also reported.

5.4 ComparisonMethods

Since our methods belong to the metric-based FSL methods, we will mainly compare our methods with metric-based methods, such as Matching Net [Vinyals et al., 2016], ProtoNet [Snell et al., 2017], RelationNet [Sung et al., 2018], IMP [Allen et al., 2019], CovaMNet [Li et al., 2019b] and DN4 [Li et al., 2019a]. Moreover, representative meta-learning based FSL methods are also listed for reference, including Meta LSTM [Ravi and Larochelle, 2017], MAML [Finn et al., 2017], SNAIL [Mishra et al., 2017], MTL [Sun et al., 2019], TAML-Entropy [Jamal and Qi, 2019], and MetaOptNet-RR [Lee et al., 2019]. Note that meta-learning based methods are essentially different from metric-based methods at two aspects. The first aspect is that an additional parameterized meta-learner is usually learned in meta-learning based methods while the metric-based methods do not have. The second aspect is that during test, metalearning based methods will fine-tune the model (or classifier) to obtain the final classification results while metric-based methods do not need fine-tuning.

Most results of these compared methods are quoted from their original work or the relevant reference. Some methods are not in the same setting with our method, such as ProtoNet, so we use the results of their modified versions to ensure fair comparison. For some recent meta-learning based methods, such as SNAIL, MTL and TAML-Entropy, we only report their results with a similar embedding network, e.g., Conv32F, which has the same architecture with Conv-64F but has 32 filters in each convolutional block.

5.5 Ablation Study

In this section, we first verify the validity of our argument on asymmetric measure for metric-based FSL. Next, based on two distribution-level measure functions, we evaluate the effectiveness of the proposed CMS strategy. Specifically, both the 2-Wasserstein distance (Wass for short) and KL divergence (KL for short) are performed on miniImageNet and tieredImageNet. Also, the contrastive versions using our proposed CMS are named as Wass-CMS and KL-CMS, respectively. Moreover, two instance-level symmetric metric based methods, ProtoNet and RelationNet, are taken as baselines.

As seen in Table 1, compared to symmetric metric based methods, such as ProtoNet, RelationNet and Wass, the proposed asymmetric measure can obtain superior results. For example, on the miniImageNet, KL gains 4.49%, 2.50% and 2.67% over these methods on the 1-shot task, respectively. This verifies that an asymmetric measure is more suitable for metric-based FSL.

We can also see that the proposed CMS strategy can indeed improve the performance of distribution-based measure functions, especially on the 1-shot setting. For instance, on the tieredImageNet, Wass-CMS achieves 0.72% improvement over Wass, and KL-CMS obtains 0.95% improvement over KL on the 1-shot task. This shows that the task-aware CMS strategy does enhance the distribution-based measure functions, thanks to taking a whole view of the entire task.

5.6 Comparison with the State of the Art

Experimental results on the comparison with the state-of-theart methods are reported in Table 2, where two types of FSL methods (i.e., both meta-learning based and metric-based) are compared. Since our methods are metric-based methods, we will mainly compare our methods with other metricbased ones. Moreover, the total number of parameters of each method is also shown in the fifth column.

From Table 2, it can be seen that the proposed ADM (I2C+KL+Fusion) outperforms all the other metric-based and meta-learning based methods on both 1-shot and 5-shot settings. For example, on the miniImageNet, our ADM obtains 10.7%, 5.81%, 3.82%, 4.66%, 3.07% and 3.02% improvements over Matching Nets, ProtoNet, RelationNet, IMP, CovaMNet and DN4 on the 1-shot task, respectively. Moreover, on the tieredImageNet, our ADM achieves 5.61%, 3.87%, 3.67%, 0.73% improvements over ProtoNet, RelationNet, CovaMNet and DN4 on the 5-shot task, respectively. This verifies the effectiveness and superiority of our proposed ADM, owing to the integration of both local and global asymmetric relations.

The proposed KL and KL-CMS are also very competitive with the state-of-the-art methods. Specifically, on the 1-shot setting, KL and KL-CMS can obtain significantly improvements over the existing metric-based methods. For instance, on the miniImageNet, KL/KL-CMS gains 9.38%/9.54%, 4.49%/4.65%, 2.5%/2.66%, 3.34%/3.5%, 1.75%/1.91% and 1.7%/1.86% improvements over Matching Nets, ProtoNet, RelationNet, IMP, CovaMNet and DN4, respectively. It verifies that such kind of distribution-based asymmetric measure is more suitable for metric-based FSL.

6 Conclusion

In this study, we provide a new perspective for metric-based FSL by considering the asymmetric nature of the similarity measure and design a novel Asymmetric Distribution Measure (ADM) network to address this task. Furthermore, to make full use of the context of the entire task, we propose a Contrastive Measure Strategy (CMS) to learn a more discriminative distribution metric space. Extensive experiments on two benchmark datasets verify the effectiveness and advantages of both local asymmetric relations and global asymmetric relations in metric-based FSL.

Acknowledgements

This work was supported by the National Key R&D Program of China (2017YFB0702600, 2017YFB0702601), National NSFC (Nos. 61806092, 61673203), and Jiangsu Natural Science Foundation (No. BK20180326).