Multi-view Feature Augmentation with Adaptive Class Activation Mapping

1 Introduction

Progresses of image classification were dramatically promoted by deep convolutional neural networks. NIN [Lin et al., 2013] for the first time replaced traditional flatteningbased fully-connected (FC) layers with global average pooling (GAP), which reduces model size and enables input images of arbitrary size. Since then, GAP is widely used in combination with more complex convolutional backbones. GoogLeNet [Szegedy et al., 2015] extracts and fuses multiscale convolutional features to enrich feature representation. ResNet [He et al., 2016] introduces residual learning to facilitate optimization of deep networks, improving model performance noticeably. Later on, more variants and derivatives of ResNet are proposed to make further improvements by increasing network width [Zagoruyko and Komodakis, 2016], adopting multi-path group convolution [Xie et al., 2017], extending skip connection to dense connections [Huang et al., 2017], and introducing attention mechanism [Wang et al., 2017; Zhang et al., 2020]. Besides, networks were also specially designed for lightweight deployment by using more efficient convolution or convolutional blocks [Howard et al., 2017; Iandola et al., 2016]. However, all these advanced deep models only focus on delicate design of convolutional backbone networks, the common ground is applying GAP to obtain vectorized features which are then fed to a linear classifier head for final classification. Though effective and efficient, GAP extracts vectorized image representations simply by averaging cross the entire feature maps. In this way, the extracted image representations could be corrupted by background elements for images where the class-related object takes up only a fraction of the image. As Figure 1 displays, even for images of the same class, the scale of class-related object varies a lot in ImageNet dataset. For images with a small scale of class-related object, GAP easily extracts image representations that bias towards features of the background or class-irrelevant objects, making the input features to the subsequent linear classifier head less class-representative.

∗Contact Author

Based on this motivation, we replace GAP with a multiview feature augmentation module (MV-FeaAug), which learns to locate region of class-related object on the final convolutional feature maps, and extracts multi-view image representations by sampling diverse local features around the attended class-discriminative region. The extracted multiview local representations more precisely represent the classrelated object of the image from diverse views, the ensemble of the predictions to all these representations are used as the final prediction of the entire image. Comparison between the image classification architecture with the general GAP and our MV-FeaAug counterpart is illustrated in Figure 2.

Apart from network architecture design, data augmentation, which acts as a regularizer to reduce overfitting, is also important for image classification. In addition to the basic position and color transformations, some advanced methods were proposed in recent years. Mixup [Zhang et al., 2017] blends two images and their corresponding labels with a random proportion as training samples. Cutout [DeVries and Taylor, 2017] randomly removes a rectangular patch from images to guide the network to learn from local views. CutMix [Yun et al., 2019] randomly cuts and pastes rectangular patches among training images with their corresponding labels mixed up in proportion to patch area. Our method shares similar spirit in expanding training samples with these methods. But instead of image-level augmentation, our method extracts diverse multi-view local representations at feature space as augmented samples for subsequent classifier head, for which we call feature augmentation. Besides, our method takes full advantage of the augmented features at inference time by ensembling predictions to all the multi-view features, while such ensemble prediction property is not available for the above-mentioned data augmentation methods.

Our method also relates to "visual explanation" of CNN models. Efforts have been made to visualize the evidence of class predictions made by CNN. CAM [Zhou et al., 2016] produces class activation maps that highlight classdiscriminative regions of images with GAP and a single-layer classifier head. Grad-CAM [Selvaraju et al., 2017] extends such object localization ability to arbitrary CNN architectures by computing gradients of the logits with respect to intermediate feature maps. However, both CAM and Grad-CAM operate in a way of postprocessing since they require extraction (or computation) of weights (or gradients) associated to the target-class logit. Moreover, the attention maps produced by these two methods are conditioned on the ground truth label, which is not flexible in cases when the target image label is unknown. In this work, we propose AdaCAM that adaptively generates attention maps with simply a forward pass of the network without conditioning on the class label. The produced attention maps provide cues to sample diverse classrepresentative local features for feature augmentation.

Another topic related to our method is attention mechanism, which has been widely applied to image classification recently. SENet [Hu et al., 2018] built a Squeeze-andExcitation module to selectively enhance or depress features in channel dimension. CBAM [Woo et al., 2018] sequentially stacked two modules for channel-wise and spatial attentions respectively. SKNet [Li et al., 2019] proposed a multibranch attention module for adaptive convolution kernel size selection. GCNet [Cao et al., 2019] improved model performance by designing an efficient self-attention block that captures global context information. All these methods increase model size for incorporating additional attention modules. By contrast, our method also introduces attention mechanism by locating class-discriminative regions via AdaCAM, however, all the extra cost is an auxiliary classifier head comprised of only one 1×1 convolutional (Conv1×1) layer. Furthermore, we demonstrate that a classification network with a deeper CNN backbone could be surpassed by a network with a shallower backbone combined with our MV-FeaAug module.

2 Method

In this section, we firstly describe our method for efficiently producing label-independent attention maps, i.e., adaptive class activation mapping (AdaCAM), and then elaborate on the complete architecture of our MV-FeaAug module.

2.1 Adaptive Class ActivationMapping

We start with a brief introduction of CAM and Grad-CAM for completeness and ease of comparison.

CAM uses GAP and a single-FC-layer classifier head to produce class activation maps. Supposing the output feature maps of the CNN backbone are {Fk}Kk=1 , where Fk is the k-th channel, and {uk}Kk=1 are the units after GAP, i.e., uk = 1 HW ∑x,y Fk(x, y), where H and W denote the height and width of feature maps. The class activation map for class c is Ac = ∑k wck Fk, where wck is the weight of the subsequent FC layer that corresponds to class c for unit uk. The limitation of CAM is threefold: (i) it is less efficient for the need of extracting intermediate weights of the model; (ii) it is limited to single-layer classifier head, and thus may compromise classification performance as compared with MLP classifier head; (iii) it is conditioned on the ground truth label, which may not be known in practical applications.

Grad-CAM addresses the second limitation of CAM by backpropagating gradients with respect to convolutional feature maps. Specifically, the class activation maps for class c is Ac = ∑k αck Fk, in which αck is the gradient of f c , the logit (before softmax) of class c, with respect to Fk, i.e., αck = 1 HW ∑x,y ∂f c/∂Fk(x, y). However, Grad-CAM is still not efficient for the need of gradient computation, and also relies on the specific class label.

A key ingredient of our MV-FeaAug module is to employ class activation maps to highlight class-related regions for local features sampling, for which a good method to produce such attention maps should follow two criteria: (i) the generation process is simple and efficient; (ii) the generation process is independent of class label, since the ground truth labels are not given at inference time. Both CAM and Grad-CAM fail to satisfy these two conditions, for which we propose AdaCAM to meet the requirements.

AdaCAM aims to directly synthesize class activation maps in a feed-forward pass without conditioning on target labels. This functionality is achieved with only a slight modification of the classifier head. As illustrated in Figure 2, after feature extraction of CNN backbone, we alter the traditional classifier head structure of [GAP→MLP→ Softmax] into [MLPConv→GAP→Softmax], where MLPConv refers to consecutive Conv1×1 layers joined by non-linear activations. The Conv1×1 layers are essentially equivalent to FC layers, but have the advantage of maintaining spatial resolution of feature maps. The output of MLPConv is feature maps {Gc}Cc=1 with its channel number equalling the number of classes C, i.e., {Gc}Cc=1=MLPConv({Fk}Kk=1 ), where Gc represents the c-th channel. Then GAP and softmax operations are employed to obtain the final softmax logit vector f , i.e., f=Softmax(GAP({Gc}Cc=1 )), the objective is to minimize the cross-entropy between f and the one-hot label vector y. This architecture is equivalent to traditional MLP classifier head since MLPConv shares the same parameter space with MLP. The advantage is the efficiency in obtaining class activation maps: for an image of class c′ , the corresponding class activation map Ac′ is exactly Gc′ , the c′ -th channel of the MLPConv output (we prove this in supplementary materials). To further make the attention map independent of class label, we perform weighted sum on {Gc}Cc=1 with respect to the softmax logit vector f as an approximation, i.e., A = ∑Ci=1 Gifi, where fi is the i-th element of f . Actually, A is very close to Ac′ since softmax operation exponentially widens the gap between elements of the logit vector.

In this work, we build a separate auxiliary classifier head tailored for AdaCAM to provide attention maps for the main classifier head. Since the auxiliary classifier is not necessarily to be sufficiently accurate, we use only one Conv1×1 layer in the MLPConv part to reduce additional overhead.

2.2 Multi-view Feature Augmentation

The overall architecture of MV-FeaAug is illustrated in Figure 4. An auxiliary classifier head is built on top of F = {Fk}Kk=1 , the final feature maps of CNN backbone, to dynamically generate AdaCAM attention maps, based on which we sample multi-view local representations from F as input to the main classifier head. The final prediction at inference time is the ensemble of the predictions made by the main classifier head to all the sampled multi-view representations.

In the auxiliary classifier part, we first use a MLPConv network comprised of a single Conv1×1 layer to reduce the depth of feature maps to the class number C, i.e., G = {Gc}Cc=1=Conv1×1(F), then the softmax logit vector f is obtained after GAP and softmax operations, i.e., f=Softmax(GAP(G)), the objective function is the crossentropy between f and the one-hot label vector y:

where yi and fi are the i-th element of y and f respectively. We call this global loss as our auxiliary classifier considers only the global view of image features by using GAP. The minimization of the global loss endows the auxiliary classifier head object localization ability, which allows for dynamic generation of AdaCAM attention map A:

The attention map A shares the same spatial size as feature maps F, and highlights class-discriminative local regions of the image. The pixel intensity of A reflects the correlation of each location with the corresponding image class. Therefore, we select top K locations with the largest pixel values from A as anchor points, around which we sample multiple local regions from F for feature augmentation. Let V be the set of anchor points for an input image, S be an ordered list of the pixel coordinates of A:

where H and W denote the height and width of A, then the anchor set V is the top K subset of S:

The number of anchor points K for each input image is a hyper-parameter of the model. Each anchor point in V serves as a centroid around which we crop R multi-scale square regions on F for feature augmentation. For feature maps F with spatial size H × W, we pre-define a region size list LR = [r1, r2, ..., rR], in which ri (odd number) denotes the length of the side of the i-th square region to be cropped around each anchor point in V . Specifically, given a centroid (xc, yc) ∈ V and a region length r ∈ LR, the square region to be cropped on F is determined by the coordinates of the topleft corner (xtl, ytl) and the bottom-right corner (xbr, ybr):

The length and the elements of LR are also hyper-parameters of our method. The purpose of sampling multi-scale local regions is to (i) capture class-related objects with different and varying spatial sizes, (ii) further increase the number of the augmented features to reduce overfitting, (iii) extract features of the class-related objects from more views to improve classification robustness. In this way, there are K × R local feature maps cropped out from F for each input image, to which average pooling is applied separately, yielding the augmented vectorized features {ai}K×R i=1 as multi-view inputs to the main classifier head, as compared with the traditional single-view input GAP(F). Like most classic networks [He et al., 2016; Szegedy et al., 2015; Xie et al., 2017; Howard et al., 2017], the main classifier head comprises a single FC layer that converts the vectorized features into logit vectors {gi }K×R i=1 , i.e., {gi }K×R i=1 =FC({ai}K×R i=1 ), after which softmax is applied to obtain probability distribution vectors {si}K×R i=1 , i.e., {si}K×R i=1 =Softmax({gi }K×R i=1 ). All the augmented multi-view features are classified to the corresponding image-level label by minimizing the cross-entropy between {si}K×R i=1 and the one-hot label vector y:

where yc and si,c are the c-th element of y and si respectively. We call this local loss as the inputs to the main classifier head are multi-view local features. The total loss function is the combination of the global loss and the local loss:

We find that the final classification performance is not sensitive to tuning the weights of Lglobal and Llocal in Eq. 7, and thus assign equal weights for the two loss functions.

At inference time, the final prediction of the image is the ensembled result of the predictions to all the augmented features, i.e., the predicted class label p is:

For computational efficiency, the inputs of the entire model is actually a mini-batch of N images, the complete loss function including the batch dimension is as below:

in which the superscript (n) indicates the n-th image sample in a mini-batch. Last but not least, our MV-FeaAug module has no restriction on the main classifier head. Besides single-layer softmax classifier, the main classifier head could be more complex one with multiple hidden layers, or more robust one like prototype classifier [Yang et al., 2018], we demonstrate this in supplementary materials.

3 Experiments

3.1 Experiment Setup

We evaluate our method on the following object, scene, and action classification datasets: Caltech101, Imagenette, Corel5k, Scene15, MIT Indoor67, Stanford Action40, UIUC Event8, UCMLU, RSSCN7, and AID. The details of these datasets are described in supplementary materials. All the images of these datasets are resized to 224×224. We verify the effectiveness of our MV-FeaAug module by applying it to networks with different CNN backbones. For all CNN backbones, the resolution of the output feature maps is kept to be 14×14, upon which we sample 3×3, 5×5, 7×7, and 9×9 multi-scale local features around each anchor point, i.e., we set LR = [3, 5, 7, 9]. In all experiments, we use the Adam optimizer with β1 = 0.9 and β2 = 0.999. The initial learning rate is lr = 0.0001, the batch size is N = 64. We only use very simple data augmentation techniques by firstly flipping images horizontally with 0.5 probability, and then enlarging images to 256×256 followed by randomly cropping back to 224×224. Our model is implemented with TensorFlow and run on a single GeForce GTX 1080 Ti GPU.

3.2 Experiment Results

Firstly, we visually demonstrate that our proposed AdaCAM possesses comparable object localization ability as CAM. To this end, we train a network with a structure of [VGG16 backbone→GAP→single FC layer] to generate CAM; and train another network of [VGG16 backbone→single Conv1×1

layer→GAP] to synthesize AdaCAM. As shown in Figure 5, though generated efficiently in a feed-forward manner and independently of class label, the attention maps obtained by our AdaCAM are very close to that of CAM.

We incorporate our MV-FeaAug module to different CNN backbones as a substitute of GAP and achieve consistent and noticeable performance gains. The chosen CNN backbones are VGG16 [Simonyan and Zisserman, 2014], ResNet50 [He et al., 2016], ResNeXt50 [Xie et al., 2017], and MobileNet [Howard et al., 2017]. For 224×224 input images, the output of VGG16 backbone is 14×14 feature maps, upon which we respectively apply GAP and our MV-FeaAug (MFA) module, yielding VGG16 GAP and VGG16 MFA respectively. For ResNet50, ResNeXt50, and MobileNet backbones that contain 5 convolutional stages, the output feature maps are of 7×7 spatial size. We firstly build GAP-based models by applying GAP over them, leading to ResNet50 GAP, ResNeXt50 GAP, and MobileNet GAP. For evaluation of MV-FeaAug, since 7×7 resolution is too small to sample sufficient multi-view local features, we remove the last convolutional stage of these backbones, yielding the truncated counterparts ResNet41, ResNeXt41, and MobileNet24 that have 14×14 output feature maps, upon which we apply MVFeaAug module to build ResNet41 MFA, ResNeXt41 MFA, and MobileNet24 MFA respectively. For fair comparison between GAP and MV-FeaAug under the same backbones, we also build ResNet41 GAP, ResNeXt41 GAP, and MobileNet24 GAP by applying GAP over the truncated backbones. We set K = 50 and LR = [3, 5, 7, 9] in our MVFeaAug module. The used classifier head of all the built models in this section is a simple single-FC-layer softmax classifier, though it is demonstrated in supplementary materials that our module also brings impressive performance gains in the case of more complex classifier head.

We train 20k iterations on Imagenette, Caltech101, AID, and Corel5k, 10k iterations on Scene15, Event8, and RSSCN7, 15k iterations on UCMLU, 30k iterations on Action40, and 40k iterations on Indoor67, making sure that learning converges on all datasets. The validation accuracies achieved by different models on different datasets are reported in Table 1. It is shown that under the same CNN backbone, image classification performance could be improved by a noticeable margin by substituting our MV-FeaAug module for GAP. Furthermore, we demonstrate that GAP based models with a deeper CNN backbone could be surpassed by our MV-FeaAug based models with a truncated shallower backbone, which shows potential value of our method in model compression. Learning curves of VGG16 GAP and VGG16 MFA evaluated on the validation set of different datasets are shown in Figure 6, our MV-FeaAug module boosts performance consistently on different datasets.

Since our method relates to both attention mechanism and data augmentation, we quantitatively compare our MVFeaAug with methods of both two fields. Based on VGG16 backbone, we separately attach a SE block [Hu et al., 2018], a CBAM block [Woo et al., 2018], a GC block [Cao et al., 2019] to the end of each convolutional stage of VGG16 backbone, introducing channel-wise attention, channel-wise and spatial attention, global context self-attention to the network respectively, yielding VGG16 SE, VGG16 CBAM, and VGG16 GC backbones. Besides, we replace the last convolutional layer of each convolutional stage of VGG16 backbone with a SK block [Li et al., 2019] that combines channelwise attention with dynamic kernel-size selection, yielding VGG16 SK backbone. For SE block, CBAM block, and SK block, we set the reduction ratio r = 2. As shown in Table 2, though injecting these attention modules can improve model performance as compared to the baseline model VGG16 GAP, the effects are very limited compared with our MV-FeaAug module (i.e., VGG16 MFA), which could be due to that our method (i) explicitly attends to class-related local features; (ii) augments training samples at feature space by sampling diverse multi-view features, while the compared attention modules do not possess such object localization and data augmentation abilities. Moreover, adding these attention modules increase backbone network parameters, while our method improves performance remarkably without enlarging backbone network. In addition, as Table 2 shows, our method also noticeably surpasses classic image-level data augmentation methods, e.g., Mixup [Zhang et al., 2017], Cutout [DeVries and Taylor, 2017], and CutMix [Yun et al., 2019] (we set αmixup=0.2 and mask size to be 56×56 in these methods), since the augmented multi-view features in our module collaboratively contribute to the final prediction of the image, while these methods do not involve such ensemble learning functionality. Ablation study of the contributing factors of our MV-FeaAug module to classification performance is demonstrated in supplementary materials.

We also explore the influence of K to classification performance based on VGG16 MFA model. Results in Table 3 shows that validation accuracy improves with the increase of K, indicating that sufficient number of the augmented multiview features is important to model performance.

Based on VGG16 MFA, we select from all the sampled K × R multi-view local regions top-k ones that correspond to local features with the highest prediction confidence of the corresponding image class in the main classifier head, and map these top-k regions from feature scale to image scale as visualization of our MV-FeaAug. Results of top-1 and top-10 local regions evaluated on images of the Imagenette validation set are shown in Figure 7, we see that the augmented multi-view local features correspond to the selected local regions that basically gather around the image-classrelated objects, which demonstrates the effectiveness of our method in extracting rich class-related local features as a kind of attention-based feature-level data augmentation.

4 Conclusion

This paper proposes MV-FeaAug module that performs feature-level data augmentation by sampling multi-view class-related local features with our proposed AdaCAM. By virtue of feature augmentation and multi-view ensemble learning, our module boosts image classification performance noticeably. In future work, we will explore the potential of our method in other learning tasks such as few-shot learning.