Multi-attention Meta Learning for Few-shot Fine-grained Image Recognition

1 Introduction

Fine-grained image recognition aims to distinguish different subordinate categories belong to the same entry-level category (e.g., various bird species [Wah et al., 2011], dog [Khosla et al., 2011] species). Different subordinate categories are distinguished by subtle and local differences, which makes fine-grained image recognition more difficult than general image recognition. Most existing fine-grained image recognition methods heavily rely on large-scale annotated training samples before learning a robust classifier [Zhang et al., 2014; Xiao et al., 2015; Fu et al., 2017]. However, annotating the fine-grained sub-categories requires strong expertise. For example, accurately identifying different bird species may need assistance from ornithologists, which is significantly expensive compared with the generic object annotation. Besides, it is hard to collect well-labeled samples of some categories, e.g., endangered, rare species. How to deal with the fine-grained image recognition with limited labeled samples is a desirable research problem in computer vision and artificial intelligence community, which has not been much explored before.

Two-year-old children can distinguish different generic categories after seeing a few images of them [Smith and Slone, 2017], while they may be confused about fine-grained categories with limited samples, as illustrated in Figure 1. Recently, few-shot image recognition has been widely studied [Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018] to identify novel category with only one or a few samples. However, few-shot image recognition for fine-grained categories has not been well studied in depth. In this paper, we focus on the problem of few-shot fine-grained image recognition (FSFGIR).

Most existing few-shot image recognition approaches [Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017; Munkhdalai et al., 2018] mainly focus on general concepts to learn generic knowledge with global representations of images. But the global representation cannot capture subtle local differences of images effectively, which may hinder these few-shot image recognition approaches from dealing with FSFRIR well.

On the other hand, most of fine-grained image recognition systems [Zhang et al., 2014; Xiao et al., 2015] follow the pipeline of finding foreground object or object parts and extracting their discriminative features. More recent work merges the pipelines into an end-to-end learning framework [Zheng et al., 2017; Fu et al., 2017; Luo et al., 2019]. However, these methods could be not employed directly for FSFGIR, since plenty of training images are unavailable in the few-shot learning. Inspired by the studies of the fine-grained image recognition, our idea is to learn discriminative parts with a small amount of training images in an end-to-end fashion, which has not been explored before.

In this work, we propose a multi-attention meta-learning (MattML) method, which leverages attention mechanisms of the base learner and task learner to capture discriminative parts of images. To be more specific, the base learner consists of a feature embedding network, two convolutional block attention modules CBAM [Woo et al., 2018] and a classifier. The two CBAM can focus on diverse and informative parts by blending cross-channel and spatial information. In the task learner, a recurrent encoder is used to learn task representations with a recurrent decoder in the auto-encoding framework, and the task representations are employed by weight generator to attend the initialization of classifier of base learner. In this way, the attended classifier obtains a task-related sensitive initialization. For adaptation, we leverage the gradient-based meta-learning approach to adjust the parameters of two CBAM and the attended classifier, such that the updated base learner can be adaptive to focus on discriminative parts according to the current few-shot task.

Our main contributions are summarized as follows:

• To our best knowledge, we first combine attention mechanisms and meta learning for FSFGIR.
• We propose a MattML method, which uses attention mechanisms of the base learner and task learner to capture discriminative parts of images.
• We establish comprehensive benchmarks for FSFGIR, and experimental results demonstrate state-of-the-art performance under the 1-shot setting.

2 Related work

2.1 Fine-grained image recognition

In the early study of fine-grained image recognition, some works [Chai et al., 2013; Xie et al., 2013] are proposed with part-based annotations of object available at both training and inference phase. Benefiting from the the development of deep neural networks, the research of fine-grained image recognition is shifted from where part-based annotations of object are known [Zhang et al., 2014] to where they are unknown [Xiao et al., 2015]. When it comes to the unknown part-based annotations, there are two main research lines. The first line is to regularize feature learning by exploiting structural relationships between fine-grained labels such as intermediate concepts [Wang et al., 2015; Xie et al., 2015] or shared attributes [Zhou and Lin, 2016]. Another line of research first localizes discriminative parts and then extracts features from these parts in a multi-stage learning framework [Xiao et al., 2015; Zhang et al., 2016]. Recently, this line of research combines part localization and feature learning in an end-to-end framework [Fu et al., 2017; Zheng et al., 2017; Luo et al., 2019]. In these end-to-end frameworks, the discriminative parts are captured by the attention mechanism, which has become a feasible approach to fine-grained image classification. As only image-level labels are available in our FSFGIR, we learn from those endto-end approachs and further propose a multi-attention metalearning method.

2.2 Few-shot image recognition

As an early attempt, Fei-Fei et al. propose a variational Bayesian framework for one-shot image classification [FeiFei et al., 2006], and Lake et al. [Lake et al., 2015] propose Hierarchical Bayesian Program Learning on the few-shot alphabet recognition tasks. Recently, there are two main lines of research to deal with the few-shot image recognition problem. The first line, named metric-based few-shot learning method, learns a transferable embedding network or function to transform images into the embedding space. And in this space the images can be recognized with a nearest neighbor [Vinyals et al., 2016; Snell et al., 2017] or a deep nonlinear metric [Sung et al., 2018]. The second line uses metalearning methods, which consists of two main components - a base learner (an initial model) and an adaptation approach (updating strategies). The base learner can be implemented with a standard network, and the adaptation approach may be implemented with parameterized networks or non-parametric strategies. The parameterized network is used to augment additive weights [Munkhdalai and Yu, 2017] or modify activation values [Munkhdalai et al., 2018] on the base learner. The typical non-parametric strategy is fixed learning rules of gradient descent [Finn et al., 2017], named gradient-based meta learning. It has become an important meta-learning approach and has been widely studied [Lee and Choi, 2018; Yoon et al., 2018].

However, these existing few-shot image recognition approaches mainly explore general object images under the fewshot learning setting. They rarely consider intrinsic properties of images, such as fine-grained categories, which are distinguished by some subtle and local differences. To this end, Wei et al. [Wei et al., 2019] introduce the FSFGIR task and propose piece-wise classifier mapping method with a bilinear network. To more effectively capture nuanced features, Huang et al. [Huang et al., 2019] present low-rank pairwise alignment bilinear network. Different from these two works, we focus on the subtle and local differences of images in the FSFGIR task by using gradient-based meta-learning approach with multi-attention mechanisms.

3 Preliminaries

3.1 Problem formulation

In the few-shot learning scenario, each problem is defined on tasks T ∼ p(T ). Each task is defined as Ti = {DTi,S,DTi,T} , where DTi,S is a support set (training samples) and DTi,T is a target set (test samples). For few-shot classification problem, the support set DTi,S = {(xi,j,yi,j) | j = 1, 2, ..., ns} and the target set DTi,T are sampled from the same distribution sharing the same label space in each task. Sampling from training, validation and test data, respectively, the training, validation and test tasks have the same forms but with disjoint label space. If the support set contains K labeled examples for each of C unique classes in the test task, it is called C-way K-shot classification problem.

3.2 Meta-learning paradigm

In practice, a meta-learning method usually learns a meta learner, which consists of two main components - a base learner (an initial model) BΘ and an adaptation approach (updating strategies) Aφ. Then the goal of meta learning is to learn an optimal meta learner across a variety of tasks to generalize to novel tasks. The training process of a meta-learning algorithm contains three alternative operations, which are: i) Task sampling: A mini-batch of tasks TB is sampled from the task distribution p(T ). ii) Task-specific adaptation. Given a task Ti ∈ TB, the initial model leverages a small set of training samples (i.e., DTi,S) to obtain update information (e.g., loss, gradient), which is further utilized by the adaptation approach to obtain the updated model BΘi . iii) Meta training. This process aims to minimize the expected empirical loss over the target set DTi,T using each task-specific updated parameters Θi across all sampled tasks. Concretely, this can be thought of learning over a collection of tasks,

where L() is a meta loss function such as the cross entropy loss for classification problems. For the test of a metalearning algorithm, provided with a new task with a small number of training samples, task-specific parameters are obtained by process ii) and can be used in the test process.

3.3 Gradient-based meta learning

Here we give an overview of the representative algorithm, model-agnostic meta learning (MAML) [Finn et al., 2017]. Analogously, MAML also contains the above mentioned two components and has the similar training process. But the adaptation approach Aφ is fixed learning rules of gradient descent (non-parametric strategies). In the task-specific adaptation, task-specific parameters Θi are obtained with one or a few gradient steps computed with loss L(DTi,S ; Θ). For onestep gradient descent, it is computed as Eq. 2, where α is a fixed learning rate of adaptation, and ∇ΘL(DTi ,S;Θ) is the corresponding gradient with respect to Θ.

In the meta training, parameters Θ are optimized on the summation of sampled tasks, which is formalized as Eq. 3. (one gradient step as exemplary)

4 The proposed method

We firstly present the architecture of base learner and task learner. And then task-specific adaptation approach is introduced in gradient-based meta-learning paradigm. Finally, we provide the formulation of meta-training objectives.

4.1 Base learner

The architecture of the base learner is shown in Figure 2, which consists of a feature embedding network, two CBAM [Woo et al., 2018] and a classifier. Next, we review CBAM.

CBAM. Each CBAM contains a channel attention module (CAM) and a spatial attention module (SAM), which are connected in tandem. Given a feature map F ∈ RC×H×W of an image, CAM produces a channel attention by exploiting inter-channel relationship of features. CAM firstly aggregates spatial information of F by using both average-pooling and max-pooling operations, which generate two channel descriptors Fcavg ∈ RC×1×1 and Fcmax ∈ RC×1×1 , respectively. As each channel of a feature map is considered as a feature detector [Zeiler and Fergus, 2014], CAM uses Fcavg and Fcmax to generate 1D channel attention map Mc ∈ RC×1×1 , i.e.,

where θc are parameters of 2 layer perceptron (P) and σ() denotes the sigmoid function. Then refined feature map F′ ∈ RC×H×W is F′ = F ⊙Mc, where ⊙ denotes elementwise multiplication and values of Mc are copied on spatial dimension. SAM generates a spatial attention by utilizing inter-spatial relationship of features. SAM aggregates channel information of F′ by using both average-pooling and max-pooling operations, and then leverages pooled features Fsavg ∈ R1×H×W and Fsmax ∈ R1×H×W to a 2D spatial attention map Ms ∈ R1×H×W, i.e.,

where θs are parameters of a layer convolution (Conv), and C(, ) denotes concatenation. Then the final refined feature map F′′ ∈ RC×H×W is F′′ = F′ ⊙Ms, where values of Ms are copied on channel dimension.

The original work [Woo et al., 2018] uses only a CBAM with a residual operation, while we use two CBAM without residual operations. And we experimentally verify our structures are better. The feature embedding network is a convolutional neural network with parameters θf and the classifier is a fully connected layer with parameters θcls. Thus the parameters of base learner BΘ contain θf , θcls, θc,k and θs,k (k = 1, 2), i.e., Θ = {θf , θcls, θc,1, θs,1, θc,2, θs,2}.

4.2 Task learner

We propose a task embedding to attend the initialization of classifier of base learner. Specifically, we introduce a task embedding network to learn task representations, which are used by weight generator to attend the initialization of parameters of classifier. Next, we introduce the task embedding network and weight generator.

Task embedding network. Task embedding learning is important to meta-learning, especially for models trained on a sequence of tasks sampled from different and disjoint distributions. It is meaningful to describe and learn relations between tasks since meta-learning models can generalize well on new tasks if they have learned some related tasks. The main characteristics of task representation learning are reflected in the following two aspects: i) huge representational capacity. ii) permutation invariance of samples. These two aspects lead to task representation learning more challenging than representation learning of examples. Inspired by common practices in learning sentence embeddings [Conneau et al., 2017], we tackle the challenge by aggregating representations of all examples, and use random order of samples to make permutation insensitive.

Given a task Ti with the support set DTi ,S = {(xi,j,yi,j) | j = 1, 2, ..., ns}, each sample xi,j is fed into an embedding network to obtain features E(xi,j). In this paper, this embedding network shares parameters with the feature embedding network and is added two extra convolutional layers. Then features of all samples in the support set are sequentially fed into the recurrent auto-encoder. The reconstruction loss is:

where ∀j , gi,j = RNNenc(E(xi,j), gi,j-1) and RNNdec(gi,j) represents the learned encoding representation and the reconstruction of the j-th example, respectively. Here RNNenc() and RNNdec() stand for a recurrent encoder (e.g., LSTM, GRU) and a recurrent decoder, respectively. The task embedding is aggregated over representations of all examples, i.e.,

where FC() represents a fully-connected layer.

Weight generator. The task embedding ei is used to generate gates of classifier ocls, i.e.,

where θcls t are parameters to predict ocls. The initial weights of classifier are θcls := ocls ⊙ θcls.

4.3 Task-specific adaptation

For a specific task Ti, the task-specific parameters Θi are obtained with one or a few gradient steps computed with the loss L(DTi,S ; Θ). But the θf are fixed. For one-step gradient descent, this process is computed as follows:

Where αcls, αk are learning rates of adaptation. Since finegrained image has subtle differences in each category, the span of updating CBAM should be smaller compared with classifier. Thus αk should be set a smaller value. Since the settings of αcls and αk may be not optimal, we set these learning rates to be learnable. The task-specific parameters are Θi = {θf , θicls , θic,1 , θis,1 , θic,2 , θis,2 }.

4.4 Meta-training objectives

Recalling the objectives for a meta-learning algorithm, we reach the optimization problem:

where Φ represents all learnable parameters including the parameters of BΘ, parameters of task learner, learning rates of adaptation (i.e., αcls, αk), and ξ is used to balance the importance of two items. The ξ is set 0.01 in the experiment. The Lr() measures the reconstruction error as defined in Eq. 6, and L() is defined in Eq. 3.

5 Experiment

5.1 Datasets

Our experiments are conducted on four fine-grained benchmark datasets (i.e., CUB Birds [Wah et al., 2011], Stanford Dogs [Khosla et al., 2011], Stanford Cars [Krause et al., 2013], FGVC Aircraft [Maji et al., 2013]). Splitting categories of these datasets forms corresponding FSFGIR datasets. For CUB Birds, Stanford Dogs and Car, we follow the splits of training, validation and test categories in [Li et al., 2019b] (i.e., FS-Birds, FS-Dogs, FS-Cars). For FGVC Aircraft, we randomly split categories to form corresponding FSFGIR dataset (i.e., FS-Aircrafts). The detailed splits of training, validation and test categories and the number of categories/images in each dataset are presented in Table 1.

5.2 ComparisonMethods

Meta-learning methods. As our method belongs to the meta-learning branch, we mainly compare meta-learning models, including MAML [Finn et al., 2017], adaCNN [Munkhdalai et al., 2018].

Metric-learning based methods. Besides meta-learning models, three well-known metric-learning based models (i.e., Matching Net [Vinyals et al., 2016], Prototypical Net [Snell et al., 2017], and Relation Net [Sung et al., 2018]) are also picked for reference. We reset the Prototypical Nets with the same 5-way training setting instead of 20-way training setting in the original work for a fair comparison. In addition, three recent state-of-the-art models (i.e., CovaMNet [Li et al., 2019b], DN4 [Li et al., 2019a], LRPABN [Huang et al., 2019]) are also compared. In these methods, Matching Net and Prototypical Net describe images with global representations, and Relation Net, CovaMNet, DN4 and LRPABN learn local descriptors of images.

5.3 Implementation details

For a fair comparison with state-of-the-art methods, we use a widely adopted CNN-4 [Vinyals et al., 2016; Snell et al., 2017] as a feature embedding network, which consists of four convolutional layers. Each convolutional layer is devised with a 3×3 convolution and 64 filters followed by batch normalization, a ReLU non-linearity and a 2 × 2 max-pooling. The input of this network is 84 × 84, and the final feature of our method for classifying is 3200-dimensional. We apply standard data augmentation, which includes random crop, left-right flip, and color jitter at the meta-training stage in all implemented experiments.

The results are reported with mean accuracy (MA) + 95% confidence intervals (CIs) over sampled 2000 tasks. And each task contains C = 5 classes, each of which has K ∈ {1, 5} examples in support set and 15 examples in target set. During training, all of models are trained from scratch in an end-toend manner across on tasks. The batch size of task is set to 4, and each task has the same settings with the above test. We use Adam optimizer [Kingma and Ba, 2015] with initial learning rate 0.001. The total iterations are 80,000 and the learning rate is changed to 1/2 after each 20,000 iterations.

5.4 Experimental results

Table 2 presents 5-way mean accuracy of different methods on FS-Aircrafts. We implement five compared methods (i.e., Matching Net, Prototypical Net, Relation Net, MAML, adaCNN) with the corresponding public code. It can be observed that our method shows absolute advantages compared with the five methods under both 5-way 1-shot and 5shot settings. Especially for the 5-way 1-shot task, the proposed MattML gains 21.28%, 18.07%, 11.75%, 15.45% and 17.09% improvements over Matching Net, Prototypical Net, Relation Net, MAML and adaCNN, respectively. For the 5-way 5-shot task, the MattML achieves 16.19%, 14.80%, 10.01%, 10.99% and 13.41% gains over Matching Net, Prototypical Net, Relation Net, MAML and adaCNN, respectively. The proposed MattML defeats the five methods by average 16.73%, 13.08% gains under 1-shot and 5-shot settings, respectively, which demonstrates the superiority of attention based meta learning method. And this also reflects few-shot learning methods of global representations do not deal with FSFGIR well.

The 5-way mean accuracy of different methods on FSDogs, FS-Cars and FS-Birds are shown in Table 3. We also implement three compared methods (i.e., Relation Net, MAML, adaCNN) with the corresponding public code on the three datasets. Similarly, the proposed MattML obtains significant improvements compared with Matching Net, Prototypical Net, Relation Net, MAML and adaCNN under both 1-shot and 5-shot settings. Compared with methods of learning local descriptors (i.e., CovaMNet, DN4 and LRPABN), MattML shows obvious superiority. Especially for the 1-shot task, MattML achieves 5.74%, 4.60% and 2.66% gains over the best one of them on FS-Dogs, FS-Cars and FS-Birds, respectively. For the 5-shot task, the proposed MattML also obtains state-of-the-art performance on FS-Dogs, and a comparable performance on FS-Birds. This indicates that it is a feasible way for FSFGIR to use attended local descriptors rather than all local descriptors.

†: reported by [Li et al., 2019b; Li et al., 2019a]

5.5 Further Analysis

Visualization. Figure 3 illustrates an example of 5-way 1shot task on FS-Dogs. In this task, 5 training samples (support set) are in the top, and test samples (target set) are in the bottom. Test samples are presented with Grad-CAM visualizations [Selvaraju et al., 2017]. In these visualizations, highlighted image regions are relevant to parts of dog (e.g., head, tail, leg, body). Specifically, salient part in the first column is head, and salient part in the fourth column is body. Interestingly, these highlighted parts can be used to distinguish their categories. This can illuminate that the proposed method can be adaptive to focus on discriminative parts with a few of training images.

Ablation Studies. Table 4 shows ablation studies of the 5-way 1-shot setting on FS-Dogs. The baseline method does not employ any attention modules (CBAM and task learner). It can be observed that: i) Only using one CBAM, we gains

3.83% improvements, which illustrates the attention mechanism (CBAM) is very useful for FSFGIR. But using residual CBAM achieves slight gains. The possible reason is the importance of CBAM has been weakened with the residual operation. ii) Compared with one CBAM, the method of using two CBAM obtains further improvements, which indicates that the proposed technique can capture complementary discriminative parts. iii) On this basis, our method achieves about 0.7% improvements with a task learner, which explains that the task learner learns a better initialization of classifier according to the current task.

6 Conclusion

In this paper, we comprehensively investigate the problem of FSFGIR. By analyzing the characteristics of fine-grained images, we propose a MattML method, which uses attention mechanisms of the base learner and task learner to capture discriminative parts of images according to the current task. Experimental results of FSFGIR on four benchmarks (i.e, FSBirds, FS-Dogs, FS-Cars, FS-Aircrafts) show the effectiveness and the superiority of the proposed method.

Acknowledgements

This work was supported by National Key Research and Development Project of New Generation Artificial Intelligence of China, under Grant 2018AAA0102500, in part by the National Natural Science Foundation of China under Grant 61532018, in part by Beijing Natural Science Foundation under Grant L182054 and Z190020, in part by National Program for Special Support of Eminent Professionals and National Program for Support of Top-notch Young Professionals, in part by the Lenovo Outstanding Young Scientists Program.