---
title: "MemREIN: Rein the Domain Shift for Cross-Domain Few-Shot Learning"
authors:
  - "Yi Xu"
  - "Lichen Wang"
  - "Yizhou Wang"
  - "Can Qin"
  - "Yulun Zhang"
  - "Yun Fu"
doi: "10.24963/ijcai.2022/505"
url: "https://ijcai.org/proceedings/2022/505"
pdf: "https://ijcai.org/proceedings/2022/0505.pdf"
venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence"
year: 2022
pages: "3636-3642"
track: "Main Track"
keywords:
  - "Computer Vision: Transfer, low-shot, semi- and un- supervised learning"
  - "Machine Learning: Multi-task and Transfer Learning"
  - "Machine Learning: Few-shot learning"
  - "Knowledge Representation and Reasoning: Applications"
  - "Knowledge Representation and Reasoning: Other"
---

# MemREIN: Rein the Domain Shift for Cross-Domain Few-Shot Learning

Yi Xu, Lichen Wang, Yizhou Wang, Can Qin, Yulun Zhang, Yun Fu

*Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 3636-3642. DOI: [10.24963/ijcai.2022/505](https://doi.org/10.24963/ijcai.2022/505)

## Abstract

Few-shot learning aims to enable models generalize to new categories (query instances) with only limited labeled samples (support instances) from each category. Metric-based mechanism is a promising direction which compares feature embeddings via different metrics. However, it always fail to generalize to unseen domains due to the considerable domain gap challenge. In this paper, we propose a novel framework, MemREIN, which considers Memorized, Restitution, and Instance Normalization for cross-domain few-shot learning. Specifically, an instance normalization algorithm is explored to alleviate feature dissimilarity, which provides the initial model generalization ability. However, naively normalizing the feature would lose fine-grained discriminative knowledge between different classes. To this end, a memorized module is further proposed to separate the most refined knowledge and remember it. Then, a restitution module is utilized to restitute the discrimination ability from the learned knowledge. A novel reverse contrastive learning strategy is proposed to stabilize the distillation process. Extensive experiments on five popular benchmark datasets demonstrate that MemREIN well addresses the domain shift challenge, and significantly improves the performance up to 16.43% compared with state-of-the-art baselines.

## Full text

## 1 Introduction

In recent years, machine learning especially deep learning methods have made amazing achievements in the field of computer vision, image classification, semantic segmentation, etc. However, the promising performance heavily counts on the large amount of well-labeled training data, which provides comprehensive and diverse samples to cover all corner cases. Such a huge scale makes it difficult in real practice, thus leads to a new topic of few-shot learning [Wang et al., 2020]. Few-shot learning aims to enable models generalize to new categories (query instances) with only limited labeled samples (support instances) from each category.

Among existing few-shot learning methods, metric-based methods have attracted significant attention because of their effectiveness and intelligibility. In general, the core idea of this kind of methods is to make classifications based on the similarity between the query images and the support images via proposed similarity measures. It usually consists of two main components: (1) feature encoder and (2) metric function. Given a task with few labeled images (support set) and unlabeled images (query set), the visual features are firstly extracted via the feature encoder and then passed through the defined metric function to determine the categories of the query images. The underlying assumption is that both training and testing are from the same dataset, namely the same domain. While, when it comes to different domains, the generalization ability of the metric-based methods greatly decreases [Chen et al., 2019; Tseng et al., 2020]. However, such ability to generalize to unseen domains is of great importance in practice, e.g., expensive human annotation or timeconsuming data collection. As a result, considering the domain shift scenario within the few-shot learning has become an important yet challenging task.

Various unsupervised domain adaptation methods have been proposed [Yang et al., 2018; Xu et al., 2022] recently. These methods aim to minimize the domain gap either by learning domain-invariant representations via representation learning, projection learning, or adversarial strategies [Long et al., 2015; Kumar et al., 2018; Tzeng et al., 2017; Kundu et al., 2019]. However, these methods assume that the complete unlabeled samples from the target domain are accessible while training. We argue that this assumption may not hold in real situations, and it could leads to high computational cost in testing phase. Domain shift problem could be addressed by various domain generalization methods [Blanchard et al., 2011; Muandet et al., 2013]. However, these methods assume that the source and target domains share the same categories. In contrast, our goal is to recognize novel categories from the target domain with only a few (e.g., 1 or 5) of samples selected from novel categories.

As argued above, there are two main challenges in crossdomain few-shot learning task. (1) How to minimize the discrepancy between the source and target domain. (2) How to recognize novel/unseen classes with only limited samples.

To this end, we propose a novel MemREIN approach, which includes Memorized, Restitution, and Instance Normalization as crucial modules, to "rein" the domain shift level in few-shot scenario. The core idea of MemREIN is to enhance the generalization ability while still be able to balance the discrimination ability for subsequent classification. Specifically, on the training stage, we first present an instance normalization layer operating on features with respect to samples at the channel level. This operation aims to reserve spatial feature dependency and meanwhile remove the image-specific features, i.e., alleviate the discrepancy of these training samples. In this way, the generalization ability across different samples is enhanced. Then, the filtered out features are extracted from a residual structure. Normally, the filtered out features are considered as useless feature which could be discarded. However, we consider it still contains finegrained distinctive knowledge which could be "remembered" and "restituted". To this end, we manage to adaptively distill the long-term discriminative information from them via our proposed novel memorized approach. Then, such discriminative information is restituted to the above refined features to maintain the discrimination ability for subsequent classification. A novel reverse contrastive loss constraint in the restitution phase is designed to encourage the better separation of discriminative features and general features, which ensures the distillation process. Contributions of our work are as,

- A novel memorized and restitution strategy is proposed for discriminative information distillation. It is able to distill the long-term discriminative information from filtered out features to maintain the discrimination ability of original features for better classification.
- An instance normalization strategy is adopted to alleviate the the discrepancy across training samples, which reduces the sample-specific features and greatly enhances the overall generalization ability across features.
- A novel reverse contrastive loss is proposed to encourage the better separation of discriminative and general features, which is able to ensure the distillation process.

Our MemREIN approach is simple yet effective. It is a universal and method-agnostic approach that can be applied to various existing metric-based methods for enhancing their generalization ability to unseen domains. Extensive experiments demonstrate the effectiveness of MemREIN, which achieves consistent superior performance than existing stateof-the-art methods under the cross-domain setting.

## 2 RelatedWork

## 2.1 Cross-domain Few-shot Learning

Few-shot classification aims to recognize novel classes with a limited amount of labeled samples. Among these existing methods, metric-based methods have attracted significant attention and achieved promising performance. For instance, MatchingNet [Vinyals et al., 2016] utilizes cosine similarity with an attention Bi-LSTM for classification and ProtoNet [Snell et al., 2017] applies Euclidean distance for classification. RelationNet [Sung et al., 2018] uses convolutional neural networks and GNN [Satorras and Estrach, 2018] uses the graph convolutional framework as the metric function.

Although these methods are able to achieve promising performance, they always fail to generalize to unseen domains since the distributions among different domains have huge shifts. Recent work [Chen et al., 2019] reveals that the accuracy of existing few-shot learning methods degrades significantly under the cross-domain setting. The motivation of our work aims to enhance the generalization ability of metricbased few-shot learning methods so that these methods can better generalize to unseen domains.

Recently, promoted by the pioneer work [Chen et al., 2019], cross-domain few-shot learning problem has attracted many attentions. As an emerging task, work [Chen et al., 2019] carried out a broader study and introduced a new benchmark. Some methods [Tseng et al., 2020; Sun et al., 2021; Phoo and Hariharan, 2020; Zou et al., 2021; Islam et al., 2021; Liang et al., 2021; Fu et al., 2021; Wang and Deng, 2021] have been proposed recently and achieved promising performance under this benchmark, which greatly promoted the development. In our work, we propose a simple yet effective method from the perspective of feature level, which is a universal and method-agnostic approach.

## 3 Method

## 3.1 Preliminaries

In the few-shot classification problem, a task T is characterized as Nw way and Ns shot, which represents the number of categories and the number of labeled samples in each category. At each iteration, the metric-based few-shot learning method randomly samples Nw categories as a task T, and then constructs a support set S = {(Xs, Ys)} and a query set Q = {(Xq, Yq)}, where X and Y represent samples and labels respectively. These two sets are constructed by randomly selecting Ns and Nq samples for each of the Nw categories.

Once the data is prepared, the feature encoder E first extracts features of the samples from both support set S and query set Q. Then, the defined metric function M predicts the query samples Xq based on three parts: the label of support samples Ys, encoded query image E(Xq), and the encoded support images E(Xs), which is formulated as,

<!-- formula-not-decoded -->

After all, the objective of the metric-based few-shot learning method is the classification loss of the samples in the query set, which is formulated as,

<!-- formula-not-decoded -->

In this paper, we tackle the cross-domain few-shot classification problem. Given a set of few-shot classification tasks T = {T1, T2, ..., Tn} as a domain (dataset). At the training stage, given N accessible domains {T seen 1 , T seen 2 , ..., T seen N }, we aim to learn a metric-based few-shot learning model with these seen domains, then the model can generalize to an unseen domainT unseen .

## 3.2 MemREINMethod

The core idea of our MemREIN method is to enhance the generalization ability, including the ability to balance the discrimination of metric-based few-shot learning methods, and achieve promising performance on arbitrary unseen domains. The overall framework is illustrated in Figure 1.

Figure 1: Framework of our MemREIN method, where ⊕, ⊖, and ⊗ denote element-wise addition, subtraction, and multiplication, respectively. With instance normalization approach, the sample-specific features of F can be reduced, and then with memorized and restitution approach, the long-term discriminative information can be distilled and restituted to refined features.

![Image](https://ijcai.org/proceedings/2022/fulltext/505/image_000000_378656ce7cc1d3276d0f18ac8e66017e34cd6a2b2e9dde223b74128e7a91aaac.png)

## Instance Normalization

As argued above, images with the same category from different domains have large discrepancies in many aspects e.g. , image style, color, quality. Generally speaking, the discrepancy between the source domain and the target domain hinders the generalization ability of the model to some extent.

To this end, we reduce the discrepancy cross samples by instance normalization in our proposed MemREIN method. Given the input feature map F ∈ Rc×h×w and the output feature map ˜ F ∈ Rc×h×w , where c, h, and w represent the number of channel, height, and width, respectively. The instance normalization is formulated as,

<!-- formula-not-decoded -->

where µ(·) and σ(·) denote the mean and standard deviation calculated at the channel level for each sample, γ ∈ Rc and β ∈ Rc are two trainable parameters. Instance normalization was originally used in style transfer [Dumoulin et al., 2016], which is helpful to enhance the generalization ability by reducing the feature dissimilarity. It can remove instance/sample specific features out of the input, which makes more general features remained.

However, instance normalization inevitably removes some discriminative information from original feature maps [Jin et al., 2020], which weakens the extracted features discrimination ability of extracted features. To address this emerging problem, we propose a memorized restitution approach to distill the discriminative information from the filtered out features and then restitute it as the final output feature maps.

## Memorized Restitution

As discussed above, in order to maintain the discrimination ability of the refined features, we propose a following memorized restitution approach to distill discriminative information. We first obtain the filtered out feature R via a residual structure, which is defined as,

<!-- formula-not-decoded -->

where R ∈ Rc×h×w , denoting the features that we have filtered out via the instance normalization operation. Since instance normalization operation will inevitably remove discriminative information from the original features, we need to distill and purify discriminative features from the residual feature R to maintain the discrimination ability.

At the training stage, given the feature map R at each iteration (we omit the subscript of feature map R for brevity), we assume R consists of two parts: D ∈ Rc×h×w with relatively more discriminative information, and G ∈ Rc×h×w with relatively more general information, which is defined as,

<!-- formula-not-decoded -->

where k represents the kth channel, θk denotes the learnable parameters to split the residual feature map R into D and G, which are channel-wise mutually exclusive.

Then, we employ the SE-like channel attention [Hu et al., 2018] to define an attention vector α = [α1, α2, ..., αc] as,

<!-- formula-not-decoded -->

where avepooling represents the average pooling layer, W1 and W2 are parameters to be learned, δ and η are the ReLU and Sigmoid activation function, respectively.

Since there are limited labeled samples under the few-shot learning framework, it is highly possible that the model would overfit. Thus we further propose a memorized mechanism with a memory vector M (l) ∈ Rc to store the long-term feature maps D, which is defined as,

<!-- formula-not-decoded -->

where M (l) k ∈ Rh×w , (l) represents the lth iteration, k denotes the kth channel. At the lth iteration, we concatenate the feature map D to the memory bank at the channel level, and update D as,

<!-- formula-not-decoded -->

where concat represents the concatenation operation, maxpooling represents the max pooling layer.

Once we obtain the updated feature map D, we restitute it to refined feature ˜ F as the final output ˜ F + of our proposed MemREIN method, and we also restitute the relatively unimportant feature map G with feature ˜ F as the "contaminated" feature ˜ F - for following loss optimization as,

<!-- formula-not-decoded -->

## Reverse Contrastive Loss

Apart from the conventional cross-entropy loss defined in Equation 2, we also propose a novel reverse contrastive loss Lrcl to promote the disentanglement of feature D and feature G. Instead of directly separating feature D and G, we first combine them with ˜ F defined in Equation 9 for better disentanglement. The reverse contrastive loss consists of two parts: L+rcl and L-rcl , e.g. , Lrcl = L+rcl +L-rcl . Given a minibatch Xb = {X1, ..., XN} contains N samples at the training phase, we first randomly select one anchor sample referred as Xa, and then we denote samples with the same category as the positive samples Xpos, samples with different categories as the negative samples Xneg. Note that the corresponding features of these samples are denoted with their subscripts such as ˜ Fa, ˜ Fpos, and ˜ Fneg in the following paragraphs.

We first reshape features ˜ F + and ˜ F - to the size of Rchw×1 and then pass them through one fully-connected layer following the softmax function to obtain the feature vectors ˜ f + and ˜ f -. Note that these two vectors have the same size of ∈ RK×1 . It is formulated as,

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

where W + and W - are trainable parameters with the same size of RK×chw , K is the number of classes. Then, our proposed reverse contrastive loss is defined as,

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

The goal of our proposed reverse contrastive loss is to promote the disentanglement of feature ˜ F + and feature ˜ F -, where feature ˜ F + contains more discriminative information and ˜ F - contains more general information. Since feature ˜ F + has better discrimination ability, sample features with the same category are closer and those with different identities are farther apart. Therefore, we propose L+rcl to encourage the features of positive samples ˜ f + pos to gather closer and separate the features of negative samples ˜ f + neg from the anchor feature. On the other hand, feature ˜ F - is more general which is not capable of distinguishing samples with the same category. Therefore, we propose L-rcl to separate the the features of positive samples ˜ f - pos from both negative sample features ˜ f - neg and the anchor feature ˜ f - a . The whole loss is defined as,

<!-- formula-not-decoded -->

where λ is a hyper-parameter to control the balance of these two terms in the training phase.

## 4 Experiments

## 4.1 Experimental Setup

Baselines. Extensive experiments are made on three existing metric-based few-shot learning methods: MatchingNet [Vinyals et al., 2016], RelationNet [Sung et al., 2018], and GNN [Satorras and Estrach, 2018]. We compare our proposed method with following existing cross-domain few-shot learning methods: FT [Tseng et al., 2020], LRP [Sun et al., 2021], and ATA [Wang and Deng, 2021] to demonstrate the advantages of our proposed method.

Datesets. Five widely used datasets are used: miniImageNet [Ravi and Larochelle, 2017], CUB [Wah et al., 2011], Cars [Krause et al., 2013], Places [Zhou et al., 2017], and Plantae [Van Horn et al., 2018].

Setting. We take the same leave-one-out setting which is applied in other baselines. Specifically, we select one dataset among CUB, Cars, Places, and Plantae as the target domain for testing, and using the remaining three datasets along with dataset mini-ImageNet as the source domains for training.

Implementation details. We adopt the ResNet-10 [He et al., 2016] as the backbone network for our feature encoder E. We insert our proposed MemREIN method after the last batch normalization layer of all the residual blocks in the feature encoder E at the training stage. Instead of optimizing from the scratch, we apply a strategy that pre-trains the feature extractor by minimizing the standard cross-entropy classification loss on the 64 training categories from the dataset mini-ImageNet and this strategy is also applied in all the baselines. In the training phase, we set λ = 0.1 and train 1000 trials for all the methods. In each trial, we randomly sample Nw categories with Ns randomly selected images for each support set, and 16 images for the query set. We use the Adam optimizer with the learning rate 0.001.

## 4.2 Experimental Results

## Quantitative Results

Table 1 shows the results under the leave-one-out setting. We first select out one dataset as the unseen domain for testing and use the remaining three datasets as well as the dataset mini-ImageNet for training since we already use the dataset mini-ImageNet for pre-training. Note that the baseline [Tseng et al., 2020] has two different training strategies, one is the "learn to learn" strategy and another is using fixed hyper-parameters. We consider the better results for comparison here, which is denoted as "+LFT" in the Table 1. The results demonstrate that our proposed MemREIN method can greatly improve the performance of all three metric-based few-shot learning methods, which reflects that our method has the capability of mitigating the domain gap problem. In addition, results show that our method consistently outperforms the "+LFT" method, which validates that our proposed method can better capture the variation of feature distributions across multiple domains than the "+LFT" method, thus the generalization ability of extracted features are better enhanced. In particular, for dataset Cars (5-way 1-shot), the accuracy of our MNet+MemREIN is 35.65 ± 0.45%, which outperforms the accuracy of baseline MNet+LFT by 16.43%.

Table 1: Classification accuracy (%) of 5-way 1/5-shot tasks under the leave-one-out setting.

| 5-way 1-shot                     | Classification Accuracy (%) - CUB   | Classification Accuracy (%) - Cars   | Classification Accuracy (%) - Places   | Classification Accuracy (%) - Plantae   |
|----------------------------------|-------------------------------------|--------------------------------------|----------------------------------------|-----------------------------------------|
| MNet [Vinyals et al., 2016]      | 37.90 ± 0.55%                       | 28.96 ± 0.45%                        | 49.01 ± 0.65%                          | 33.21 ± 0.51%                           |
| MNet+LFT [Tseng et al., 2020]    | 43.29 ± 0.59%                       | 30.62 ± 0.48%                        | 52.51 ± 0.67%                          | 35.12 ± 0.54%                           |
| MNet+MemREIN (Ours)              | 46.37 ± 0.50%                       | 35.65 ± 0.45%                        | 54.92 ± 0.64%                          | 38.82 ± 0.48%                           |
| RNet [Sung et al., 2018]         | 44.33 ± 0.59%                       | 29.53 ± 0.45%                        | 47.76 ± 0.63%                          | 33.76 ± 0.52%                           |
| RNet+LFT [Tseng et al., 2020]    | 48.38 ± 0.63%                       | 32.21 ± 0.51%                        | 50.74 ± 0.66%                          | 35.00 ± 0.52%                           |
| RNet+MemREIN (Ours)              | 52.02 ± 0.52%                       | 36.38 ± 0.38%                        | 54.82 ± 0.57%                          | 36.74 ± 0.45%                           |
| GNN [Satorras and Estrach, 2018] | 49.46 ± 0.73%                       | 32.95 ± 0.56%                        | 51.39 ± 0.80%                          | 37.15 ± 0.60%                           |
| GNN+LFT [Tseng et al., 2020]     | 51.51 ± 0.80%                       | 34.12 ± 0.63%                        | 56.31 ± 0.80%                          | 42.09 ± 0.68%                           |
| GNN+MemREIN (Ours)               | 54.26 ± 0.62%                       | 37.55 ± 0.50%                        | 59.98 ± 0.64%                          | 45.69 ± 0.64%                           |
| 5-way 5-shot                     | Classification Accuracy (%)         | Classification Accuracy (%)          | Classification Accuracy (%)            | Classification Accuracy (%)             |
|                                  | CUB                                 | Cars                                 | Places                                 | Plantae                                 |
| MNet [Vinyals et al., 2016]      | 51.92 ± 0.80%                       | 39.87 ± 0.51%                        | 61.82 ± 0.57%                          | 47.29 ± 0.51%                           |
| MNet+LFT [Tseng et al., 2020]    | 61.41 ± 0.57%                       | 43.08 ± 0.55%                        | 64.99 ± 0.59%                          | 48.32 ± 0.57%                           |
| MNet+MemREIN (Ours)              | 67.31 ± 0.51%                       | 47.36 ± 0.48%                        | 68.14 ± 0.58%                          | 52.28 ± 0.52%                           |
| RNet [Sung et al., 2018]         | 62.13 ± 0.74%                       | 40.64 ± 0.54%                        | 64.34 ± 0.57%                          | 46.29 ± 0.56%                           |
| RNet+LFT [Tseng et al., 2020]    | 64.99 ± 0.54%                       | 43.44 ± 0.59%                        | 67.35 ± 0.54%                          | 50.39 ± 0.52%                           |
| RNet+MemREIN (Ours)              | 68.39 ± 0.48%                       | 46.92 ± 0.50%                        | 69.87 ± 0.54%                          | 58.64 ± 0.50%                           |
| GNN [Satorras and Estrach, 2018] | 69.26 ± 0.68%                       | 48.91 ± 0.67%                        | 72.59 ± 0.67%                          | 58.36 ± 0.68%                           |
| GNN+LFT [Tseng et al., 2020]     | 73.11 ± 0.68%                       | 49.88 ± 0.67%                        | 77.05 ± 0.65%                          | 58.84 ± 0.66%                           |
| GNN+MemREIN (Ours)               | 77.54 ± 0.62%                       | 56.78 ± 0.66%                        | 78.84 ± 0.66%                          | 65.44 ± 0.64%                           |

Table 2: Performance study on the hyper-parameter λ.

| 5-way 5-shot - GNN+MemREIN   | Classification Accuracy (%) - CUB   | Classification Accuracy (%) - Cars   |
|------------------------------|-------------------------------------|--------------------------------------|
| λ = 0.01                     | 77.02 ± 0.62%                       | 56.12 ± 0.66%                        |
| λ = 0.1                      | 77.54 ± 0.62%                       | 56.78 ± 0.66%                        |
| λ = 0.5                      | 77.34 ± 0.62%                       | 56.66 ± 0.66%                        |
| λ = 1                        | 76.78 ± 0.64%                       | 56.22 ± 0.66%                        |

## Qualitative Results

As illustrated in Figure 2, we employ the t-SNE algorithm to visualize features that obtained by the feature encoder "before/within/after" our MemREIN method, where each color represents one class. We take the GNN baseline under the leave-one-out setting on the dataset CUB as the example. We randomly select 5 categories with 60 samples of each category in the testing spilt of the dataset CUB. The first column indicates two examples of the features from conventional GNN baseline, The second column indicates the features that only applied the instance normalization operation, and the third column indicates the features that applied our proposed MemREINmethod. As shown in the first column, there exists several rough clusters but the boundaries are unclear. After instance normalization, the overall model generalization ability of features is enhanced. In comparison with the first column and the third column, the features learned by our method are more clustered and separable, which validates the effectiveness of our novel memorized restitution approach.

Figure 2: t-SNE visualization of features extracted by encoder.

![Image](https://ijcai.org/proceedings/2022/fulltext/505/image_000001_212f6f0f1146f032883cf8da59a67d1ebb929df312180ff2a9b5dcd7a4b61ab6.png)

Table 3: Ablation study on our method. "GNN+IN" indicates that we only employ the instance normalization strategy, "w/o L-rcl " indicates that we remove the L-rcl term, and "w/o L+rcl " indicates that we remove the L+rcl term, "GNN+MemREIN w/o MB" represents that we remove memory bank and directly use the feature map D, and "GNN+MemREIN (D&amp;G)" represents that the memory bank is operated both on feature map D and G (not shared).

|   5-way 5-shot - Variant ID | Classification Accuracy (%) - Method   | Classification Accuracy (%) - CUB   | Classification Accuracy (%) - Cars   | Classification Accuracy (%) - Places   | Classification Accuracy (%) - Plantae   |
|-----------------------------|----------------------------------------|-------------------------------------|--------------------------------------|----------------------------------------|-----------------------------------------|
|                           1 | GNN [Satorras and Estrach, 2018]       | 69.26 ± 0.68%                       | 48.91 ± 0.67%                        | 72.59 ± 0.67%                          | 58.36 ± 0.68%                           |
|                           2 | GNN+IN                                 | 67.34 ± 0.66%                       | 42.76 ± 0.75%                        | 67.82 ± 0.73%                          | 54.04 ± 0.69%                           |
|                           3 | GNN+MemREIN (Ours)                     | 77.54 ± 0.62%                       | 56.78 ± 0.66%                        | 78.84 ± 0.66%                          | 65.44 ± 0.64%                           |
|                           4 | w/o L-rcl                              | 75.38 ± 0.63%                       | 55.34 ± 0.72%                        | 78.03 ± 0.68%                          | 65.22 ± 0.64%                           |
|                           5 | w/o L+rcl                              | 73.02 ± 0.62%                       | 51.45 ± 0.64%                        | 73.26 ± 0.66%                          | 62.22 ± 0.64%                           |
|                           6 | GNN+MemREIN w/o MB                     | 75.98 ± 0.62%                       | 54.64 ± 0.66%                        | 74.86 ± 0.68%                          | 64.08 ± 0.68%                           |
|                           7 | GNN+MemREIN (D&G)                      | 76.02 ± 0.66%                       | 55.26 ± 0.69%                        | 78.08 ± 0.66%                          | 64.84 ± 0.68%                           |

Table 4: Classification Accuracy (%) of our proposed method with different Nw. We consider the CUB dataset as the unseen domain under the leave-one-out setting.

| 5-shot                        | Classification Accuracy (%) - 2-way   | Classification Accuracy (%) - 5-way   | Classification Accuracy (%) - 10-way   | Classification Accuracy (%) - 20-way   |
|-------------------------------|---------------------------------------|---------------------------------------|----------------------------------------|----------------------------------------|
| MNet [Vinyals et al., 2016]   | 78.46 ± 0.78%                         | 51.92 ± 0.80%                         | 38.22 ± 0.38%                          | 26.17 ± 0.24%                          |
| MNet+LFT [Tseng et al., 2020] | 83.88 ± 0.72%                         | 61.41 ± 0.57%                         | 45.69 ± 0.39%                          | 32.81 ± 0.23%                          |
| MNet+MemREIN (Ours)           | 88.68 ± 0.68%                         | 67.31 ± 0.51%                         | 49.22 ± 0.34%                          | 33.99 ± 0.22%                          |
| RNet [Sung et al., 2018]      | 84.25 ± 0.72%                         | 62.13 ± 0.74%                         | 47.15 ± 0.40%                          | 34.52 ± 0.24%                          |
| RNet+LFT [Tseng et al., 2020] | 85.44 ± 0.72%                         | 64.99 ± 0.54%                         | 49.90 ± 0.40%                          | 37.20 ± 0.25%                          |
| RNet+MemREIN (Ours)           | 89.12 ± 0.66%                         | 68.39 ± 0.48%                         | 52.85 ± 0.32%                          | 42.82 ± 0.20%                          |

## Performance Study of λ

We carry out performance study on the hyper-parameter λ. We take our method under the leave-one-out setting (5-way 5-shot) and dataset CUB and Cars as the example. We set four different values λ = {0.01, 0.1, 0.5, 1} and the results are shown in Table 2. It can be observed that when setting λ = 0.1, it can achieve the best performance.

## Ablation Study

We carry out ablation studies of different components in our proposed method. We compare with the GNN baseline under the leave-one-out setting (5-way 5-shot) and results are shown in Table 3. Comparing the results of Variant 1 and 2, it indicates that only applying the instance normalization operation results in the decrease of the accuracy. It is reasonable because the instance normalization operation will inevitably remove some discriminative useful information. In comparison with Variant 3 and 6, it validates the effectiveness of employing the memory bank on feature D. Comparing Variant 3, 6, and 7, it indicates that when employing memory bank on feature G, it would cause performance decrease. Empirically, when applying the memory bank on the feature D and directly using feature G, it can achieve the best performance.

## Different Numbers of Ways

We consider a more practical situation that Nw may be different from that at the training stage. It also reflects the generalization ability of the model and results are shown in Table 4. Note that model GNN requires the number of ways to be the same while the training and testing, thus we evaluate with method MatchingNet and RelationNet (MNet and RNet for short). The model is trained on the datasets mini-ImageNet, Cars, Places, and Plantae and evaluated on the dataset CUB with different number of ways Nw. The results indicate that our proposed method are still capable of improving the generalization ability to the unseen domain with various numbers of ways. In addition, our proposed method consistently outperforms the baseline that has considered the domain-shift issue, which validates the superiority of our method.

## 5 Conclusion

In this paper, we investigated the cross-domain few-shot classification problem where exists the domain gap issue. We propose a novel framework, MemREIN, which considers Memorized, Restitution, and Instance Normalization to address this issue. We first alleviate feature dissimilarity across sample features via an instance normalization algorithm to enhance the overall generalization ability. In order to avoid the loss of fine-grained discriminative knowledge between different classes, a memorized restitution approach is further proposed to adaptively remember the long-term refined knowledge and restitute the discrimination ability. Finally, a novel reverse contrastive learning strategy is proposed to stabilize the distillation process. Extensive experiments demonstrate that MemREIN well addresses the domain shift challenge, and significantly improves the performance up to 16.43% compared with state-of-the-art baselines.

## References

1. [Blanchard et al., 2011] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifi- cation tasks to a new unlabeled sample. In NeurIPS, 2011.
2. [Chen et al., 2019] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR, 2019.
3. [Dumoulin et al., 2016] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2016.
4. [Fu et al., 2021] Yuqian Fu, Yanwei Fu, and Yu-Gang Jiang. Meta-fdmixup: Cross-domain few-shot learning guided by labeled target data. In ACM MM, 2021.
5. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
6. [Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In CVPR, 2018. [Islam et al., 2021]
7. Ashraful Islam, Chun-Fu Chen, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, and Richard J Radke. Dynamic distillation network for cross-domain few-shot recognition with unlabeled data. In NeurIPS, 2021.
8. [Jin et al., 2020] Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo Chen, and Li Zhang. Style normalization and restitution for generalizable person re-identification. In CVPR, 2020.
9. [Krause et al., 2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In ICCV Workshops, 2013.
10. [Kumar et al., 2018] Abhishek Kumar, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogério Schmidt Feris, Bill Freeman, and Gregory W Wornell. Co- regularized alignment for unsupervised domain adapta- tion. In NeurIPS, 2018.
11. [Kundu et al., 2019] Jogendra Nath Kundu, Nishank Lakkakula, and R Venkatesh Babu. UM-Adapt: Unsuper- vised multi-task adaptation using adversarial cross-task distillation. In ICCV, 2019.
12. [Liang et al., 2021] Hanwen Liang, Qiong Zhang, Peng Dai, and Juwei Lu. Boosting the generalization capability in cross-domain few-shot learning via noise-enhanced super- vised autoencoder. In ICCV, 2021.
13. [Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
14. [Muandet et al., 2013] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via in- variant feature representation. In ICML, 2013.
15. [Phoo and Hariharan, 2020] Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across ex- treme task differences. In ICLR, 2020.
16. [Ravi and Larochelle, 2017] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017. [Satorras and Estrach, 2018] Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In ICLR, 2018.
17. [Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.
18. [Sun et al., 2021] Jiamei Sun, Sebastian Lapuschkin, Wo- jciech Samek, Yunqing Zhao, Ngai-Man Cheung, and Alexander Binder. Explanation-guided training for cross- domain few-shot classification. In ICPR, 2021.
19. [Sung et al., 2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learn- ing. In CVPR, 2018.
20. [Tseng et al., 2020] Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In ICLR, 2020.
21. [Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
22. [Van Horn et al., 2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inatural- ist species classification and detection dataset. In CVPR, 2018.
23. [Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim- othy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NeurIPS, 2016.
24. [Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. 2011.
25. [Wang and Deng, 2021] Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversar- ial task augmentation. In IJCAI, 2021.
26. [Wang et al., 2020] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few ex- amples: A survey on few-shot learning. ACM Computing Surveys, 2020.
27. [Xu et al., 2022] Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn. arXiv:2203.05046, 2022.
28. [Yang et al., 2018] Baoyao Yang, Andy J. Ma, and Pong C. Yuen. Learning domain-shared group-sparse representa- tion for unsupervised domain adaptation. Pattern Recog- nition, 2018.
29. [Zhou et al., 2017] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 2017.
30. [Zou et al., 2021] Yixiong Zou, Shanghang Zhang, Jianpeng Yu, Yonghong Tian, and José MF Moura. Revisiting mid- level patterns for cross-domain few-shot recognition. In ACM MM, 2021.