From Pixels to Objects: Cubic Visual Attention for Visual Question Answering

1 Introduction

Visual Question Answering (VQA) is an interdisciplinary research problem, which has attracted extensive attention recently [Yang et al., 2016; Lu et al., 2016; Hyeonseob Nam and Kim, 2017; Lu et al., 2018]. It has the potential to be applied for assisting the visually impaired people and automatically querying on large-scale image or video datasets [Xu and Saenko, 2016; Hyeonseob Nam and Kim, 2017]. Compared with image captioning task, the VQA task requires a deeper understanding of both image and question rather than a coarse understanding [Goyal et al., 2017;

∗Corresponding author: Lianli Gao, Heng Tao Shen

Spatial Visual Attention Cubic Visual Attention

Figure 1: The illustration of spatial visual attention and cubic visual attention. Spatial visual attention is modelled as a weight matrix on the last conv-layer feature map of a CNN encoding an input image. Different channels share the same weights. Instead, cubic visual attention learns both spatial and channel attention, which conforms to the nature of conv features, i.e., spatial and channel-wise.

Antol et al., 2015]. It inspects intelligent system's ability by inferring a correct answer for the visual question.

Existing works for VQA can be generally classified into two categories: 1) Typical CNN-RNN models, which transfer image captioning frameworks by integrating CNN with Recurrent Neural Networks (RNN) to solve VQA tasks [Gao et al., 2015; Ren et al., 2015; Malinowski et al., 2015]; and 2) Question-guided visual attention mechanisms, which aim to discover the most important regions to answer a question by exploring their relationships [Fukui et al., 2016; Lu et al., 2016; Hyeonseob Nam and Kim, 2017; Shih et al., 2016; Xu and Saenko, 2016; Yang et al., 2016]. More specifically, question-guided visual attentions are conducted by concatenating the semantic representation of a question with each candidate region and then put them into a multiple layer perceptron (MLP) or applying the dot product of each word embedding and each spatial location's visual feature [Xu and Saenko, 2016]. In addition, Yang et al. [Yang et al., 2016] proposed a stacked attention model by utilizing semantic representation of a question as query to search for the regions in an image multiple times to infer an answer progressively. In [Hyeonseob Nam and Kim, 2017], a dual attention was introduced to infer the answers by attending to specific regions in images and words in text. Lu et al. [Lu et al., 2016] presented a co-attention that jointly performs question-guided visual attention and image-guided question attention to address 'where to look' and 'what words to listen to'.

Although promising results have been achieved, typical CNN-RNN models resort to a global image presentation which may contain noisy or unnecessary information for the related question. To some extent, question-guided visual attention mechanisms have tackled this problem generally by spatial attention, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, such spatial attention does not necessarily conform to the attention mechanism-a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial and channel-wise [Chen et al., 2017]. Image features are generally extracted by deep Convolutional Neural Networks (CNNs) [He et al., 2016; Song et al., 2017; Gao et al., 2017]. Starting from an input color image of the size W × H × 3, the last convolutional layer consisting of C-channel filters output a W ′ × H′ × C feature map. Different channels of a feature map is essentially activation response maps of the corresponding filter, and channel-wise attention can be viewed as the process of selecting semantic attributes on the demand of the sentence context. For example, when we want to predict cat, our channel attention (e.g., in the conv5 3/conv5 4 feature map) will assign more weights on channel-wise feature maps generated by filters according to the semantics like furry texture, ear, and cat-like shapes. Channel attention plays a different role compared with spatial attention, and it is rarely addressed in previous works.

In this paper, we take the full advantage of two characteristics (i.e., channel and spatial) of object-based region features for visual attention-based VQA. Specifically, we propose a novel Cubic Visual Attention (CVA) framework by successfully applying a channel attention and a spatial attention to assist VQA task. An object detection network [Ren et al., 2017], which has the potential to enable nearly cost-free object region proposal, is applied to extract top-k objects in an image and each object is represented by a D-dimensional vector. Next, the channel-wise attention learns to pay attention to specific channels of the last conv feature map. Thirdly, a region-based spatial attention is applied on the channelattended features to select related objects. Finally, an answer is inferred by considering both the attended visual features and the question.

2 Cubic Visual Attention for VQA

The VQA task is to predict an answer from a question and a related image. In this section, we introduce our proposed CVA framework (shown in Fig. 2) for VQA, and it consists of 1) a feature extraction component, which extracts the features for question and input image; 2) a channel attention for selecting filters related to the high-level object semantic; 3) a region-based spatial attention for learning to focus on the regions of the image that are important and 4) an answer prediction layer to infer to answer. In this section, we describe each of them.

2.1 Input Representations

Image Features. For the input image, we use an existing state-of-the-art object detection network Faster R-CNN [Ren et al., 2017] for object proposal and feature extraction. Specifically, each image is input into Faster R-CNN model to obtain the top-K candidate objects. For each selected region/box k, vk is defined as the mean-pooled convolutional feature from this region and each vk indicates a vector with D dimensions. Therefore, the input image V is defined as follow:

EncodingQuestionFeatures. Suppose that [q1 , q2 , ..., qT ] represents an question, where qt is an one-hot representation for the word at position t, and T is the length of the question. Each word representation qt is transferred into a lower dimensional vector xt with an embedding matrix W qe .

There are various approach to encode a question like bagof-words. However, Long Short-term Memory (LSTM) and gated recurrent unit (GRU) are two of the most popular mechanisms to encode a sentence in machine translation and great results have been obtained. In this paper, we employ GRU to encode our questions, and the gradient chains of GRU do not vanish due to the length of questions. For the t-th time step, the GRU unit takes the embedding vector xt as an input, updates the gate zt, resets gate rt, and then outputs a hidden state ht. After T steps, we obtain the T-th output hT to represent the semantic information of a question. We formulate our encoding process as below:

where Wz,r,h, Uz,r,h, bz,r,h are the parameter which needed to be learn. Note that σ is a sigmiod activation function, and ◦ is used as the Hadamard product or element-wise multiplication. Through the question feature encoding process, we generate the question representation Q, which equals to the last output of GRU (hT ).

2.2 Channel Attention

In this section, we introduce a novel channel-wise attention mechanism to attend the visual features V. Each vi is obtained by mean pooling the conv features of a given box spatial location. Essentially, each channel of a feature map in CNN is correlated to a convolutional filter which performs as a pattern detector. For instance, the lower-level filters detect visual clues such as edges and color, while the higher-level filters detect semantic patterns, such as attributes or object components. In this work, our object region features are pooled from the last conv feature map, thus each channel represents the semantic patterns of the detected objects within an image. Therefore, conducting a channel-wise attention can be viewed as a process of choosing object semantic attributes.

For channel-wise attention, we first reshape V to U and U = [u1, u2, ..., uD], where ui ∈ Rk represents the i-th dimension of the whole object feature V , and D is the dimension of vi or it is the total number of channels for each object region. Next, we apply a mean pooling for each channel to generate the channel vector

where ui is the mean vector of ui, which represents the i-the channel features. Our channel-wise attention Ac is defined as below:

where Wvc, Wqc and Wc are embedding matrices, bvc, bqc and bc are bias terms, and ⊗ indicates the outer product of vectors. To sum up, we obtain our channel-wise attention weight β through our channel-wise attention Ac, defined as follow:

2.3 Object Region-based Spatial Attention

With above step, we obtain the channel-wise attention weight β, thus we can feed β to a channel-wise attention function fc to calculate a modulated feature map Vc :

where fc is an channel-wise multiplication for region feature map channels and corresponding channel weights. In addition, eventually Vc is

Suppose, we have Vc where vci indicates the visual feature of the i-th object region. In general, a question may only relate to one or several particular regions of an image. If we want to ask 'what the color of the dog', then only the dog object region contains the useful information, therefore typical CNN-RNN which employing the whole global visual feature may lead to sub-optimal results due to the irrelevant visual regions shown in the input image. Instead of considering each object region equally, our region-based spatial attention mechanism aims to target the most related region with an referred question. Given the previous calculated Vc , a singlelayer neural network is adopted to take both Vc and Q as inputs to generate a new feature a, and then a softmax function is followed to compute the region-based spatial attention weight η. The object region-based spatial attention As is defined as :

where Wvo and Wqo are the embedding matrices that project both visual and question features into a common latent space. In addition, Wo is a set of parameters that needs to be learn. b is the model bias. ⊕ is the addition of a matrix and a vector. Moreover, η ∈ Rk is a k-dimensional vector, which represents the importance of each object region. Therefore, the weights η can be calculated by:

Furthermore, to deal with multiple object regions, a simple strategy usually is used to compute the average of features across the whole image, and this generated feature is used as input to integrate question feature to generate an answer:

However, as mentioned above this strategy effectively effectively integrate multiple regions into a single vector, neglecting the inherent spatial structure and leading to the loss of information. Instead of using above simple strategy, we apply η to attend where to look at and it is defined as bellow:

2.4 Answer Prediction

Following previous work [Wang et al., 2017; Lu et al., 2016], we treat the answer prediction process as a multi-class classification problem, in which each class corresponds to a distinct answer. We predict the answer based on the stacked attended image visual feature Vs and a question feature Q, and a multilayer perceptron (MLP) is used for classification:

whereWv,Wq,Wh are parameters. bh, bp are bias terms and p is the probability of the final answer.

2.5 AVariant of CVA

The previous introduced stacked attention mechanism applies channel-wise attention before spatial attention. Given an initial question feature Q and visual region features V, we adopt a channel-wise attention Ac to compute the channel-wise attention weights β for obtaining a channel-wised weighted object region features Vc . Next, we apply the object region based spatial attention As by taking Vc as inputs to obtain region spatial weights η. The pipeline of this framework can be summarized as follows:

In order to further study the effect of the oder of channelwise and spatial attentions, we propose an CVA variant which exchange the order of two attentions by firstly applying spatial attention As and then following by a channel-wise attention Ac. This pipeline can be summarized as follows:

where Us is obtained by reshape Vs , seen channel-wise reshape operation. Further more, Vc and Q are utilized to predict the final answer. For simplicity, we name this pipeline as CVA-V.

3 Experiments

3.1 Datasets

We evaluate our proposed model on three public image QA datasets: the COCO-QA dataset, the VQA dataset and Visual7W dataset.

COCO-QA dataset. This dataset [Ren et al., 2015] is proposed to enable training large complex models due to the reason that the previous DAQUAR dataset only contains approximately 1,500 images and 7,000 questions on 37 common object classes. Therefore, COCO-QA is created to produce a much larger number of QA pairs and a more evenly distributed answers based on the MS-COCO dataset. It contains in total 117,684 samples with 78,736 as training and 38,948 as testing. There are 23.29% and 18.7% overlap in training questions and training question-answer pairs. COCO-QA

consists of four question categories: Object (54,992 Training vs 27,206 Testing), Number (5,885 vs 2,755), Color (13,059 vs 6,509) and Location (4,800 vs 2,478). In addition, all answers in this dataset are a single word.

VQA dataset. It is a large-scale dataset presenting both open-ended answering task requiring a free-form response and multiple-choice task requiring an approach to pick from a predefined list of possible answers. Specifically, 204,721 real images (123,287 training and validation vs 81,434 testing ) are collected from the newly-released Microsoft Common Objects in Context (MS COCO) dataset. For this VQA dataset challenge, it contains two test categories: 1) testdev for debugging and validation purpose; and 2) test standard indicates the 'standard' test data for the VQA competition. More specifically, we use 248,349 training questions, 121,512 validation questions and 244,302 test questions. The question types include Y/N 38.37%, Number 12.31% and Other. Compared with COCO-QA, the answer form is diversity with an answer containing one, two or three words respectively being 89.32%, 6.91%, and 2.74%. In addition, for each image, three questions are collected and each question is answered by ten subjects alone with confidence. Following [Hyeonseob Nam and Kim, 2017], we set th number of possible answer for VQA as 2,000.

Visual7W. This dataset is collected recently by Zhu et al [Zhu et al., 2016] with 327,939 QA pairs on 47,300 COCO images, which is a subset of Visual Genome image dataset. With the Amazon Mechanical Turk (AMT), they collected 1,311,756 human-generated multiple-choices and 561,459 object groundings from 36,579 categories. For each QA pair, it has four human-generated multiple-choices and only one of them is correct. In addition, for Visual7W, it consists of seven types of question including what,where,when, who,why,how and which. The first six type of questions are proposed to examine the capability of a model of visual understanding. Compared with VQA, Visual7W contains rich questions and longer answers. Moreover, Visual7W establishes an explicit link between QA pairs and image regions by providing complete grounding annotations and providing diverse question to acquire detailed visual information. Following [Yu et al., 2017], we only test our model in the settings of multiplechoices.

3.2 EvaluationMetrics

The VQA task is usually regarded as a multi-class classification problem, and thus accuracy is an important evaluation metric for evaluating the performance of VQA models. Following [Antol et al., 2015], we use the following equation to compute the classification accuracy:

where ans is the answer predicted by a VQA model.

In addition, for MS COCO dataset, we also report the performance in terms of the Wu-Palmer similarity (WUPS), which accounts for word-level ambiguities in the answer words. The equation is defined as [Malinowski et al., 2015] and it contains a thresholded taxonomy-based Wu-Palmer similarity parameter. For COCO-QA, we report WUPS at two extremes, 0.0 and 0.9.

3.3 Implementation Details

For extracting visual object features, we first integrate Faster R-CNN with ResNet-101 retrained on the ImageNet dataset by following an image captioning approach [Anderson et al., 2017] and then select top 36 (k = 36) object regions and each region is represented as 2,048 dimensional features. For sentence encoding, a pre-trained GloVe word embedding of dimension (300) and a single layer GUR are utilized. In addition, the dimension of every hidden layer including GRU, attention models and the final joint feature embedding is set as 1,024.

In our experiments, our models are trained with Adam. The batch size is set to 256, and the epoch is set as 30. More specifically, gradient clipping technology and dropout are exploited in training.

3.4 Ablation Study

For VQA challenge, it contains the test-dev, which is proposed to debug and validate VQA models, thus VQA competition evaluation server allows for unlimited submission. In this section, we perform ablation study on the VQA dataset to qualify the role of each component in our model. Specifically, we re-train our approach by ablating certain components: 1) channel-wise attention only (CA); 2) object region attention only (RA); 3) our stacked attention with both channel-wise and region-based attention (CVA); and 4) reversed stacked attention (R-CVA) by changing the order of channel-wise and region-based attention to test whether their order effect the VQA performance.

The experimental results are shown in Tab.1 (test-standard is not recommended to be used for VQA ablation study). From the experimental results, we can see that object region attention alone performs better than channel-wise attention only on the Y/N, Other and All with an increase of 2.86%, 3.46% and 2.83% respectively. In terms of Number, channelwise attention performs slightly better. By stacking those two attention into a VQA model in any order, we find that the stacked attention models improve the performance of VQA, especially for Number by approximately 1.9%. In addition, changing order would slightly effect the performance.

3.5 Comparing on the VQA dataset

Compared Methods. We compare our opposed CVA with the state-of-the-art VQA approaches, which can be divided into four categories:1) No attention approaches (LSTM Q+I [Antol et al., 2015] and deeper+norm [Antol et al., 2015] and DPPnet [Noh et al., 2016]); 2) visual attention based methods (SAN [Yang et al., 2016], FDA [Ilievski et al., 2016], DMN+ [Xiong et al., 2016], MCB+Att. [Fukui et al., 2016] and MCB-7 (ensemble of 7 Att. models) [Fukui et al., 2016]); 3) Utilizing high-level concepts as visual features or semantic attention based methods (AC [Wu et al., 2016] and ACK [Wu et al., 2016] ); 4) methods with both image attention and question attention (HieCoAtt [Lu et al., 2016] and DAN [Hyeonseob Nam and Kim, 2017]) and 5) jointly learning semantic attention and visual attention (MLAN (ResNet) [Yu et al., 2017]). Our proposed method CVA belongs to the second category, visual attention only.

Results. The experimental results are shown in Tab.2. We have the following observations: CVA obtains the best 'all'

Q:Is the laptop turned on? Our Prediction: yes Human Answer: yes

Table 4: Evaluation results on Visual7W dataset.

accuracies on test dev (Open-end 65.92% vs multiple-choice 70.3%) and test-standard (66.20% vs 70.41%). Our method belongs to the second category (visual attention only). Compared with the second category methods, the improvement over the best approach MCB is significant, by 5.12% (testdev, open-ended, all) and 4.9% (test-dev, multiple-choice, all). In addition, we also report the results of MCB-7 (ensemble of 7 Att. models), although it is not comparable because each model in ensemble of 7 models uses MCB with attention. Comparing with the third category, again our approach outperforms the best approach ACK, especially with an increase of 6.8% in terms of test-standard opened-end all. Although HieCoAtt and DAN integrate both visual and question attentions, our CVA performs better. For standard-test dataset, tt surpass the DAN by 2.0% (in open-ended, all) and 1.14% (in multiple-choice, all), respectively. In addition, compared with MLAN involving both semantic attention and visual attention, our approach is also better, with a clear performance gap on the test-dev dataset. The results in Tab.2 clearly demonstrate the advantage of our method.

3.6 Comparing on the COCO-QA dataset

Compared Methods. In this section, we compare our methods with the state-of-the art approaches on the COCOQA dataset. We compare it with 2-VIS+BLSTM [Ren et al., 2015], IMG-CNN [Ma et al., 2016], DDPnet[Noh et al., 2016], SAN(2, CNN)[Yang et al., 2016], QRU [Li and Jia,

2016] and HieCoAtt [Lu et al., 2016]. In addition, Fig.3 demonstrates some qualitative results from VQA.

Results. The experimental results are demonstrated in Tab. 3. Our CVA achieves the highest performance with an accuracy of 67.51% on all and a WUPS 0.9 of 76.7%. Compared with the best counterpart HioCoAtt and the second best counterpart QRU, the improvement of accuracy is 2.11% and 5.01% on all. Specifically, for Object, Color and Loc. three types questions, CVA increases the accuracy to 69.55% and 68.96%, and 59.93%, respectively. The table indicates that the Number type question performs worst in all methods, with approximately 10 % to 19% lower than other types. This might be caused by the unbalanced training dataset. The number of training samples of Object type is 54,992, which is nine times more than the number of training samples for Number.

3.7 Comparing on the Visual7w dataset

In this section, we further assess our model on the recent released dataset Visual7w. For the experiments, we compare our CVA with previous work LSTM-Att. [Zhu et al., 2016], MCB+Att. [Fukui et al., 2016] and MLAN [Yu et al., 2017] and the experimental results are shown in Tab.3. Both LSTM-Att. and MLAN are competitive. However, compared with LSTM-Att., MLAN utilizes a much lower fusion methods (2400-dimension vs 16,000-dimension)[Yu et al., 2017]. Tab.3 shows that our CVA achieves the highest scores. In particular, the MLAN employs both visual attention and text attention, while our CVA only exploits the visual attention.

4 Conclusion

In this paper, we propose a novel cubic visual attention network for visual question answering task. CVA takes the full advantage of characteristics of CNN to obtain visual channelwise features representing semantic attributes and object region based visual features representing rich semantic information to support visual question answering and it achieves the state-of-the-art on three public standard datasets across various question types, such as multiple-choices and openended questions. The contribution of CVA is not only provide a powerful VQA model, but also a better mechanism to understand the visual information for predicting answers.

Acknowledgments

This work is supported by the Fundamental Research Funds for the Central Universities (Grant No. ZYGX2014J063, No. ZYGX2014Z007) and the National Natural Science Foundation of China (Grant No. 61772116, No. 61502080, No. 61632007, No. 61602049).