Cross-Domain Few-Shot Classification via Adversarial Task Augmentation

1 Introduction

Few-shot classification [Lake et al., 2015] aims to classify instances from unseen classes with few labeled samples in each class. To this end, many meta-learning based models elaborately design various task-shared inductive bias (e.g., the metric function [Sung et al., 2018], the inference mechanism [Garcia and Estrach, 2018; Liu et al., 2019]) to solve few-shot classification tasks. They demonstrate promising performance when evaluated on the tasks from the same domain with the training tasks (e.g., both training and testing are on the mini-ImageNet classes). However, some works [Chen et al., 2019; Guo et al., 2020] have shown that the existing meta-learning models perform undesirably when there exists domain shift between training tasks and test tasks (e.g., training on the mini-ImageNet classes and testing on the ISIC classes), and even underperform compared to traditional pretraining and fine-tuning. As a result, the cross-domain fewshot classification problem has attracted considerable attention from the machine learning community, especially the difficult single domain generalization problem [Tseng et al., 2020; Guo et al., 2020].

∗Corresponding author

To generalize to unseen domains without accessing any data from those domains, some domain generalization models have been proposed [Volpi et al., 2018; Li et al., 2019]. They learn the classifiers that generalize to the unseen domains, and assume that the source and unseen domains share the same classes. However, in the few-shot classification problem, the classes in the target tasks are unseen before. The most similar works to ours are [Tseng et al., 2020] and [Sun et al., 2020] which aim to improve the performance of meta-learning models in cross-domain tasks. [Tseng et al., 2020] introduces the feature-wise transformation layers for the metric-based metalearning models which modulate the feature activation with affine transformation to improve the robustness of the metric functions. But as mentioned above, the different metalearning models have various inductive bias, not just the metric functions. [Sun et al., 2020] uses explanation-guided training to prevent the feature extractor from overfitting to specific classes, but it needs to manually derive the explanations for different meta-learning models.

We aim to find a method that is general, easy to implement and can improve the robustness of various inductive bias. To this end, we resort to the task augmentation techniques which constructs 'challenging' virtual tasks to increase the diversity of training tasks. For image classification, various hand-crafted data augmentation techniques (e.g., horizontal flip, random crop and color jitter) can be used for task augmentation. However, they have limited effect and cannot perform adaptive augmentation for different inductive bias. Recently, some works [Sinha et al., 2018; Volpi et al., 2018] proposed adaptive sample (e.g., images) augmentation methods to improve the robustness of the model. Inspired by these works, we propose an inductive bias-adaptive task augmentation method to improve the cross-domain generalization ability of the meta-learning models.

Concretely, we consider the worst-case problem around the source task distribution T0

where θ ∈ Θ represents the model parameters, L(T; θ) is the loss function which depends on the model's inductive bias, and D(T , T0) is the distance metric between task distributions. Compared with minimizing the loss function on the source task distribution T0, the solution to the worst-case problem (1) guarantees good performance on the wider space of task distributions which are ρ distance away from T0, as illustrated in Figure 1. By solving the worst-case problem (1), we propose a task augmentation method. Since the loss function depends on the inductive bias, our method can adaptively generate 'challenging' tasks according to the different inductive bias and increase the diversity of training tasks which improves the robustness of the model under domain shift. What's more, our method can be used as a plug-andplay module for various meta-learning models.

The main contributions of this work are as follows:

• To the best of our knowledge, this is the first work that introduces task augmentation into cross-domain fewshot classification to improve the generalization ability of meta-learning models under domain shift.
• We consider the worst-case problem around the source task distribution T0, and propose a plug-and-play inductive bias-adaptive task augmentation method, which can be conveniently used for various meta-learning models.
• We evaluate our method on the RelationNet [Sung et al., 2018], the GNN [Garcia and Estrach, 2018] and one of the state-of-the-art models TPN [Liu et al., 2019] with extensive experiments under the cross-domain setting. Experimental results show our method can significantly improve the cross-domain generalization performance of these models and outperforms [Tseng et al., 2020] and [Sun et al., 2020]. And under the same settings, the meta-learning models with our adversarial task augmentation module can outperform the traditional pre-training and fine-tuning under domain shift.

2 RelatedWork

Cross-domain few-shot classification. Although various meta-learning models for few-shot classification have achieved impressive performance, they fail to generalize to unseen domains. To this end, [Tseng et al., 2020] uses the feature-wise transformation layers to simulate various distributions of image features during training and thus improve the generalization capability of the metric function. [Sun et al., 2020] uses the explanation methods to upscale the features which are more relevant to the prediction, and penalize them more when overfitting occurs to avoid the intermediate features from specializing towards fixed classes. Different from them, we focus on improving the robustness of various inductive bias. Other models [Liu et al., 2020; Yeh et al., 2020] that appear in the CVPR 2020 Cross-Domain Few-Shot Learning Challenge use various techniques to solve crossdomain few-shot classification tasks, e.g., batch spectral regularization, model ensemble and large margin mechanism.

Domain generalization. Domain generalization methods [Volpi et al., 2018; Li et al., 2019] have been developed to generalizing from single or multiple seen domains to the unseen domains without accessing samples from them. However, these models consider the setting that the seen and unseen domains share the same categories. In contrast, in the cross-domain few-shot classification problem, the seen and the unseen domains have completely disjoint categories.

Adversarial training. Adversarial training [Goodfellow et al., 2015] aims to make deep neural networks be capable of resistant to adversarial attacks. [Sinha et al., 2018] proposes principled adversarial training through distributionally robust optimization, where virtual images are model-adaptively generated by maximize some risk and the models learned with these new images become more robust. In this work, we introduce a similar model-adaptive augmentation method into the meta-learning models, and propose a plug-and-play module to generate virtual 'challenging' tasks to improve the robustness of various meta-learning models.

3 Method

3.1 Preliminaries

Few-Shot Classification

Each few-shot classification task T consists of a support set Ts and a query set Tq. If the support set Ts contains C classes with K samples in each class, the few-shot classification task is called C-way K-shot. The query set Tq contains the samples from the same classes with the support set Ts. Formally, a few-shot task can be defined as T = (Ts, Tq), where Ts = {(xsi , ys i )}C×K i=1 and Tq = {(xqj , yq j )}Qj=1 . Given the support set Ts, our goal is to classify the samples in the query set Tq correctly to one of the C classes. Typically, the base learner A is needed to output the optimal classifier ψ of the task basing on the support set Ts, i.e., ψ = A(Ts; θ) and it depends on the inductive bias.

The main difference among meta-learning models for fewshot classification lies in the design choices for the inductive bias. For examples, the RelationNet [Sung et al., 2018] chooses the metric function based on convolutional neural

Algorithm 1 Adversarial Task Augmentation

Input: Source task distribution T0; initialized parameters θ0 Require: Learning rate α and β; iteration number for early stopping Tmax; probability of using original data p ∈ (0, 1); candidate pool of filter sizes K

Output: learned parameters θ

• 1: Initialize: θ ← θ0
• 2: while training do
• 3: Randomly sample source task T0 = (X0, Y0) from T0
• 4: X0 ←RandConv(X0,K) ( with probability 1-p )
• 5: for i = 1, . . . , Tmax do
• 6: Xi =Xi-1 +β · ∇XL((Xi-1, Y0); θ)
• 7: end for
• 8: θ ← θ -α∇θL((XTmax , Y0); θ)
• 9: end while

networks (CNNs), the GNN [Garcia and Estrach, 2018] applies generic message-passing inference mechanism on a partially observed graphical model, and the TPN [Liu et al., 2019] utilizes the transductive label propagation. Metalearning models aim to learn these inductive bias over a collection of tasks which are assumed to be sampled from the task distribution T0, and the learning objective is

where Lmeta is the loss function, such as the classification loss of the samples in the query set Tq, and θ represents the model parameters.

Cross-Domain Setting

Generally, the target tasks are assumed to come from the source task distribution T0. However, in this work we consider the few-shot classification under domain shift. Concretely, we focus on the single domain generalization problem because the data from multiple training domains may not always be available due to data acquiring budget or privacy issue. We denote the domain as the distribution of the fewshot classification tasks. The target tasks come from several unknown domains {T ′ 1 , · · · , T ′N }. The goal is to learn a metalearning model using the single source domain T0, such that the model can generalize to the several unseen domains.

3.2 Adversarial Task Augmentation

Next, we solve the worst-case problem (1) to get a plug-andplay model-adaptive task augmentation module. In order to make the loss function L(T; θ) depending on the inductive bias of the meta-learning models, inspired by Equation (2), we define it as

To allow task distributions that have different support to that of the source task distribution T0, we use the Wasserstein distances as the metric D. Concretely, for task distribution T and T0 both supported on the task space H, let Π(T , T0) denotes their couplings, meaning measures M on H2 with M(T, H) = T (T ) and M(H, T ) = T0(T ). The Wasserstein distance between T and T0 is

where d : H×H→ R+ is the transportation cost from T to T0, satisfying d(T, T0) ≥ 0 and d(T, T ) = 0.

Basing on the Proposition 1 in [Sinha et al., 2018] and the Theorem 1 in [Blanchet and Murthy, 2019], we have the following duality result.

Lemma 1. Let L : H×Θ → R and d : H×H → R+ be continuous. Let φγ(T0; θ) = supT∈H{L(T ; θ) - γd(T, T0)} be the cross domain surrogate. For any distribution T0 and any ρ > 0,

and for any γ ≥ 0, we have

Thus, the continuity of the loss function L(T; θ) and the transportation function d(T; T0) with respect to T needs to be satisfied to solve the worst-case problem (1). For this, we model the task T as the vector with the fixed dimension. A common approach is to use task embedding to model the tasks, but it is not applicable here. The reasons are as follows: 1) it conflicts with the definition of the loss function L(T; θ), i.e., calculating L(T; θ) requires the support set Ts and query set Tq, not the task embedding; 2) we expect T0 and T to be the distribution of the tasks to generate virtual tasks not the task embedding. We treat each task as the vector concatenated by the samples and labels it contains, i.e.,

where [·, ·] denotes the concatenation operation. This definition is equivalent to treating the distribution of the tasks as the joint distribution of samples and labels within the task, i.e.

Meanwhile, we assume that the number of samples in the task T is fixed, so as the dimension of T. The change of the elements of the samples in task T leads to the change of T, so the continuity of L(T; θ) and d(T; T0) can be satisfied. Another consideration for assuming a fixed number of samples in a task is that we want to generate the virtual task containing the same number of samples with source task T .

In the worst-case problem (1), the supremum over task distributions is intractable, so we consider its Lagrangian relaxation with penalty parameter γ ≥ 0

Applying Lemma 1, our optimization problem becomes

Further, applying the Theorem 4.13 in [Bonnans and Shapiro, 2013], we have the following result to solve the problem (10). Lemma 2. Let L : H×Θ → R be λ-Lipschitz smooth and d(·, T0) be µ-strongly convex for each T0 ∈ H. If γ > λµ , there is unique ˆ T satisfying

and

In the Lemma 2, γ > λµ ensures that the function L(T ; θ)- γd(T, T0) is (γµ - λ)-strongly concave in T, so that there exists the unique ˆ T .

According to Lemma 2, we can solve the Equation (11) to generate the virtual cross-domain task ˆ T and use it to update the model parameters θ

where α is the learning rate. From the Equation (11), we can make two insights: 1) for the meta-learning models, the virtual task ˆ T is more 'challenging' than the source task T0 and the loss function satisfies L( ˆ T ; θ) ≥ L(T0; θ)+γd( ˆ T , T0), so the model learned with it tends to be more robust; 2) since the loss function L(T; θ) depends on the inductive bias, solving the Equation (11) is equivalent to adaptively generating the virtual task that is more 'challenging' to the currently learned inductive bias.

For deep networks and other complex models, the supremum problem in Equation (11) cannot be solved accurately, so we use the gradient ascent process with early stopping to solve it. Concretely, let the set of all samples in a task be X and their corresponding labels be Y , i.e.,

then T = (X, Y ) and T0 = (X0, Y0). We use the source task T0 as the initialization of T, and the task vector defined in Equation (7) as the optimization variable. Considering that in different few-shot classification tasks, samples with the same labels can correspond to different real category (e.g., cat, dog), so the change of label Y is not considered here, i.e., keeping Y = Y0. In the i-th iteration, the update is

Here the regularization term -γd(T, T0) is removed from the iteration goal and the reasons are as follows: 1) this term is used to constrain the proximity of the virtual task to the source task T0, but using the source task as the initialization and early stopping can achieve the same effect, see the Section 4.3 for the detailed discussion; 2) it reduces the computational overhead and hyper-parameters requiring hand-tuning. After Tmax iterations, we get the virtual 'challenging' task ˆ T = (XTmax , Y0) and update the model parameters θ with it. See Algorithm 1 for the full description of the training process. Given an unseen task, the inference process is the same as the original meta-learning model. Note that if Tmax = 0, Algorithm 1 becomes the original meta-learning training process, so our method is a plug-and-play module.

In this paper, we mainly consider the cross-domain fewshot image classification, and the convolutional neural networks (CNNs) are the necessary tools. However, CNNs tend to overfit on superficial local textures [Geirhos et al.,

2019], so we use the random convolutions [Lee et al., 2020] that can change the local textures and keep the shape unchanged as the auxiliary augmentation technique for our adversarial task augmentation. Concretely, given an input image I ∈ RC×H×W , where H and W are the height and width and C is the number of feature channels, the filter size k is first randomly sampled from the candidate pool K, then the Xavier normal distribution [Glorot and Bengio, 2010] is used to initialize the convolution weights. The stride and padding size are determined to make the transformed image having the same size with I. In practice, for each task T0 sampled from T0, we keep its all samples unchanged with probability p, or use the same random convolution on its all samples to get a new task for training, as shown in the fourth line of Algorithm 1.

4 Experiments

In this section, we evaluate the adversarial task augmentation method on the RelationNet [Sung et al., 2018], the GNN [Garcia and Estrach, 2018] and one of the state-of-the-art meta-learning models TPN [Liu et al., 2019], and compare it with [Tseng et al., 2020] and [Sun et al., 2020]. These meta-learning models have different kinds of inductive bias so as to verify the versatility and effectiveness of our method.

4.1 Experimental Settings

Datasets. We conduct extensive experiments under crossdomain settings, using nine few-shot classification datasets: mini-ImageNet [Ravi and Larochelle, 2017], CUB, Cars, Places, Plantae, CropDiseases, EuroSAT, ISIC and ChestX, which are introduced by [Tseng et al., 2020] and [Guo et al., 2020]. Each dataset consists of train/val/test splits and please refer to these references for more details. We use the miniImageNet domain as the single source domain, and evaluate the trained model on the other eight domains. We select the model parameters with the best accuracy on the validation set of the mini-ImageNet for model evaluation.

Implementation details. In all experiments, we use the ResNet-10 [He et al., 2016] as the feature extractor and use the Adam optimizer with the learning rate α = 0.001. We find that setting Tmax = 5 or 10 is sufficient to obtain satisfactory results, and we choose the learning rate of the gradient ascent process β from {20, 40, 60, 80}. We set K = {1, 3, 5, 7, 11, 15} for all experiments and choose p from {0.5, 0.6, 0.7}. We evaluate the model in the 5-way 1-shot/5shot settings using 2,000 randomly sampled episodes with 16 query samples per class, and report the average accuracy (%) as well as 95% confidence interval.

Pre-trained feature extractor. Instead of optimizing from scratch, we apply an additional pre-training strategy as in [Tseng et al., 2020] which pre-trains the feature extractor by minimizing the standard cross-entropy classification loss on the 64 training classes in the mini-ImageNet dataset.

4.2 Evaluation for Adversarial Task Augmentation

We apply the adversarial task augmentation module to the RelationNet, the GNN, and the TPN models to evaluate its effect on improving the cross-domain generalization ability of the meta-learning models, and compare it with [Tseng et al., 2020] which adds the feature-wise transformation layers to the feature extractor and [Sun et al., 2020] which uses explanation-guided training. All models are trained and tested in the same environment for the fair comparison and the results are shown in Table 1.

We can observe that with our adversarial task augmentation module, the cross-domain few-shot classification accuracy of the meta-learning models is consistently and significantly improved. And compared with [Tseng et al., 2020] and [Sun et al., 2020], our method achieves comparable or more significant improvement, which means that adaptively enhancement of different inductive bias is more effective than enhancing artificially determined inductive bias. Moreover, applying the feature-wise transformation layers even harms the crossdomain generalization performance of the TPN model, while our method is still effective, which means that our method is more general, not just suitable for the metric-based metalearning models (the RelationNet and the GNN models).

4.3 Ablation Study

Effect of the randomconvolution. As aforementioned, we use the random convolution for auxiliary task augmentation. Here we study the effect it brings. Figure 2 shows the average few-shot classification accuracy on eight unseen domains without random convolution and that obtained by complete method. As we can see, without the random convolution, our method still improves the cross-domain generalization ability of the meta-learning models, and outperforms '+FT' [Tseng et al., 2020] and '+LRP' [Sun et al., 2020]. Using the random convolution can achieve further improvements.

Is the regularization term useful? In the Section 3.2, we remove the regularization term -γd(T, T0) from the iteration goal and here we will show it is reasonable. We consider two common candidates for distance d(T, T0) and find that they do not bring benefits. As we assumed, the label composition of few-shot classification tasks is the same as each other, so the distance between task T and T0 depends on the samples X and X0. Let the feature vectors of X and X0 are F = {f i }Ni=1 and F0 = {f i0 }Ni=1 with N = C × K + Q. The first candidate is the direct sample-wise Euclidean distance, i.e., d(T, T0) = 1N ∑Ni=1 ‖f i - f i0 ‖22 and the second candidate is the maximum mean discrepancy (MMD) distance, i.e., d(T, T0) = ‖ 1N ∑Ni=1 f i - 1N ∑Ni=1 f i0 ‖22 . Figure 3 shows the average few-shot classification accuracy on eight unseen domains without the regularization term, with the sample-wise Euclidean distance regularization term and with the maximum mean discrepancy (MMD) distance regularization term. We set the hyper-parameter γ = 1 and do not use the random convolution for the clear comparison. As we can see, using the regularization term does not bring obvious benefits or even is harmful, which shows that early stopping has already imposed enough constraints, and using the regularization term leads to excessive limits.

4.4 Comparison with Fine-tuning

[Guo et al., 2020] shows that in the cross-domain fewshot classification problem, traditional pre-training and finetuning outperform the meta-learning models. Here we reexamine this phenomenon through a different fair comparison, i.e., using data augmentation while solving an unseen task. Given an unseen task T consisting of C ×K support samples and Q query samples, for the fine-tuning, we use the pre-trained feature extractor as initialization and a fully connected layer as the classification head. For each epoch, we generate 15 pseudo samples for each class based on the support samples using the data augmentation method from [Yeh et al., 2020] and use these C × 15 pseudo samples and the support samples for fine-tuning where we use the SGD optimizer with the learning rate 0.01 and the momentum 0.9 as in [Guo et al., 2020]. For the meta-learning models, we use the parameters trained on the mini-ImageNet with our adversarial task augmentation method as the initialization and adapt the meta-learning models to the same C × (K + 15) samples as above at each iteration where the C × 15 pseudo samples are used as the pseudo query set, and we use the Adam optimizer with the learning rate 0.001. All models are fine-tuned for 30 (or 50) epochs in the 5-way 1-shot (or 5-shot) tasks. Since all models use the same amount of target domain data when solving each unseen task, it is a fair comparison. The results are shown in Table 2. As we can see, the meta-learning models with our adversarial task augmentation module significantly outperform the traditional pre-training and fine-tuning even under domain shift.

5 Conclusion

In this paper, we aim to design a new method that can improve the cross-domain generalization capability of meta-learning models in the cross-domain few-shot learning. For this, we consider the worst-case problem around the source task distribution T0, and propose a plug-and-play inductive biasadaptive task augmentation method, which significantly improves the cross-domain few-shot classification capability of various meta-learning models, and outperforms the existing works. This is the first work to achieve the above objective by generating 'challenging' virtual tasks. We also compare the meta-learning models with pre-training and fine-tuning under the same settings, and find that the meta-learning models with our method outperform the fine-tuning under domain shift.