Show, Observe and Tell: Attribute-driven Attention Model for Image Captioning

1 Introduction

Image captioning enables machines to understand an image and generate a descriptive caption for it. It is challenging due to: 1) accurately recognizing all concepts attached to the given image, including objects, attributes, relationships, etc., is problematic; and 2) teaching machines to mimic how humans verbally describe an image does not seem easy. However, this topic is of great significance to the ultimate goal of scene understanding, and thus attracts much attention from academia and industry.

∗This work was supported by the National Natural Science Foundation of China (No. 61571269), the National Natural Science Foundation of China (No. 61701273) and the Project Funded by China Postdoctoral Science Foundation (No. 2017M610897). Corresponding author: Guiguang Ding

Figure 1: Top: the proposed framework for predicting the attributes of the given image. We regard the attribute prediction as a sequential learning process and adopt the encoder-decoder framework to infer the attributes. Bottom: We obtain attribute information and its corresponding attention maps, which are used as the context features in the proposed approach. The attributes include object terms (women, umbrella), relational terms (under) and descriptive terms.

So far, many pioneering works [Karpathy and Li, 2015; Mao et al., 2015; Vinyals et al., 2015; Jia et al., 2015] have been proposed for image captioning. Early works usually adopted an encoder-decoder framework, where the information of an image is encoded into a static representation by a convolutional neural network (CNN), and a recurrent neural network (RNN), e.g., Long Short-TermMemory (LSTM), is employed as a decoder to interpret it into sentences.

In spite of great advances, these CNN-RNN captioning models exploit the word generation on the basis of the image representation directly, without explicitly taking more high-level semantic information from image into consideration [Yao et al., 2017]. Later on, it has been verified that attributes with rich semantic cues about the image are effective in captioning [Yao et al., 2017; Wu et al., 2016; Liu et al., 2017]. To integrate the attribute information into the decoder RNN, they regarded the attribute prediction as a multiple single-label classification problem and extend CNN to predict attributes independently.

However, strong co-occurrence dependencies are common among attributes. For example, sea usually appears together with wave, but rarely co-occurs with cars. On the other hand, the attributes are composed of relational terms and descriptive terms, apart from the object terms. We claim that modeling the dependencies among attributes can facilitate the inference of relational terms. For example, in Fig. 1, object terms, women and umbrella, can help to recognize the relational term, under. Since the RNN can model the relationship among elements in a sequential manner by storing the dependencies in its internal memory, it is more natural and appropriate to adopt the RNN to infer the attributes, especially for the relational terms.

Besides, it is noticed that most attention-based captioning models directly attend to the feature map obtained from the CNN. The information of the feature map can be redundant or irrespective of the content of the image due to the uniform grid of equally sized and shaped receptive fields for feature map [Anderson et al., 2017]. For example, for the image in Fig. 1, the regions of background (indicated by the red arrow) may be irrespective of the image content, and the features for regions resembling in each other (indicated by the yellow arrow), are redundant. Therefore, a more compact representation for images should be explored for attention, so that the attention results can more precisely capture the visual information related to the predicted word.

In this paper, we propose an attribute-driven attention model for image captioning. Firstly, instead of simply adopting a CNN to predict the attributes, we employ a CNN-RNN framework with the attention mechanism to predict attributes, where CNN provides feature representations and RNN acts as an inference module to predict the attributes. Secondly, we extract the context features corresponding to the attributes, as illustrated in Fig. 1, and incorporate them with the attribute information into the captioning model. In this way, the cooccurrence dependencies among attributes can be maintained in the memory of the RNN. Besides, the context features derived from the supervision of attributes are more semantically related to the image content, and thus are more compact than the feature map from CNN.

Overall, our contributions are three-fold:

• We model the co-occurrence dependencies among attributes by adopting a CNN-RNN framework and incorporating the visual attention mechanism for attribute detector.
• We filter the redundant or irrelevant features in relation to the image content by extracting the context features for attributes, and integrate themwith the attribute information into the captioning model.
• We show the effectiveness and superiority of the proposed approach by conducting a massive of experiments and comparisons with other approaches on the MS COCO image captioning dataset.

2 RelatedWork

There have been many pioneering neural network-based works for image captioning. The early neural network-based works [Mao et al., 2015; Vinyals et al., 2015] aim to explore the way of aligning the visual feature and the semantic feature. They simply compressed the image information into a single and static representation, which can lead to losing information that could be useful for richer, more descriptive captions [Xu et al., 2015]. Besides, the feature representations are extracted from a CNN directly and thus lack more high-level semantic information.

Attribute-based approaches. Attribute-based approaches aim at boosting the performance of a captioning model with the more high-level semantic information. You et al. [2016] proposed a semantic attention model to selectively attend to the most related attributes during caption generation. Liu et al. [2017] treated the attribute information as a semantic regularisation for the captioning model. Yao et al. [2017] regarded the attributes as the auxiliary information and explored different ways to incorporate them into captioning. These works achieved inspiring results by incorporating the attribute information into the captioning model. However, they simply predict the attributes with a CNN, without considering the co-occurrence dependencies among attributes. In this paper, we aim at modeling the co-occurrence among attributes via employing both the CNN and the RNN.

Visual attention models. Inspired by the presence of attention in the human visual system, Xu et al. [2015] firstly proposed soft attention and hard attention to make the decoder exposed to different aspects of image information at each time step. Chen et al. [2017a] thought that the attention should be spatial and channel-wise, and proposed SCACNN model to selectively attend to both salient regions and salient semantic patterns. Anderson et al. [2017] pre-trained an object detection model on another dataset, and used it to obtain image feature at concept level for attention. Our approach feeds the attention model with the context features corresponding to the predicted attributes, which is similar to the idea in [Anderson et al., 2017]. But unlike [Anderson et al., 2017], where an object detection model is pre-trained on a large dense captioning dataset with precise position information about the concepts in images, our model does not need a pre-trained object detection model and is trained on image captioning dataset without assistant datasets. Please note that we can also leverage other large datasets to boost the performance of the attribute prediction, which is out of the scope of this paper and is left as our future work.

Multi-label classification. Another related work is the multi-label classification domain. Our attribute prediction model is similar to [Wang et al., 2016; Liu et al., 2017]. But their works are built on the standard datasets for multilabel classification task, which means that labels of images for training are annotated by humans. While in our work, the attributes are obtained from the captions corresponding to images, which consist of not only object terms and descriptive terms but also relational terms, and thus are more diverse and more in quantity.

3 Methodology

In this section, we introduce the whole framework of the proposed approach in detail. As illustrated in Fig. 2, our model is composed of three components: the inference module, the attention module and the generation module. The inference module aims to predict the attributes and produce their observed context features in a sequential manner. The generation module attempts to generate the sentence word by word, on the basis of the image information. And the attention module interacts with the inference module and the generation module and provides different types of features, e.g., v∗ t and c∗t for them.

3.1 Encoder-Decoder Framework

We start by briefly introducing the encoder-decoder framework. Given an image I and a sequence X = {x0, x1, x2, ..., xT } which depends on the task background, e.g., attributes for multi-label classification, the encoderdecoder framework aims to predict the next element xt+1 conditioned on the image I and the partial sequence {x0, x1, ..., xt} that has been generated. Generally, a CNN is employed as the encoder to extract the features of the given image, and a RNN acts as the decoder to generate the sequence conditioned on the image features. It directly maximizes the following objective:

where the conditional probability is modeled by the decoder RNN: RNN(zt, xt, ht-1,mt-1), where ht-1 and mt-1 are the hidden state vector and memory cell vector of RNN at time step t - 1, respectively. zt is the auxiliary knowledge, like, the global image feature obtained from the last fully connected layer in CNN, or the context feature of the image from the output of the attention module.

Here we formulate the RNN function as a variant of LSTM as in [Rennie et al., 2017].

3.2 AttentionModule

The attention module plays a core role in the proposed approach. At different time steps, it captures the most related information of the image to the next prediction of the generated words. Our attention module can provide two types of image features, the region-based features for the inference module and the attribute-based ones for the generation module, respectively.

Region-based feature. At time step t, given the feature map from the CNN, i.e., V = {vi|i = 0, 1, 2, ..., k, vi ∈ RD } where each subscript i denotes the i-th image region, our attention module is able to adaptively attend to the most relevant image regions to the next prediction under the guidance of the hidden state ht-1 of the RNN in the inference module.

where Wa, Wav and Wah are parameters to be learned. We define v∗ t as the context feature corresponding to the attribute which contains the visual information of those image regions related to the attribute.

Attribute-based feature. The context feature v∗ t is more semantically related to the image content. We extract a series of context features corresponding to attributes, and when generating the next word in the generation module, the attributebased feature is derived by adaptively attending to these context features. For the ease of explanation, we substitute v∗ t with ci and use C = {c0, ..., ct, ..., cl}, i.e. ct = v∗ t , to represent the context feature map corresponding to the attributes, where l is the number of attributes. So the attribute-based feature is obtained as follows:

where Wb, Wbc and Wbh are parameters to be learned. ¯ht-1

is the hidden state of the RNN in the generation module at time step t - 1.

Our attention model can be considered as a stack attention model. The first layer of attention can be regarded as a filter, merging regions and compressing the redundant or irrelevant features, and thus the context feature corresponding to the attribute is more compact than the feature map in the CNN. The second layer of attention operates the standard attention mechanism on those compact features from the the first layer and thus can be more semantically related to the next word to be predicted.

3.3 Inference Module

The inference module aims to predict attributes for the given images, following the encoder-decoder framework. First, the information of images and attributes will be projected to the same embedding space to capture the image-text relationship. Then a RNN is adopted to take in the embedding vectors in the embedding space at each time step and maintain the label co-occurrence information in its internal memory.

Specifically, at time step t, the information of the image, v∗ t , i.e., the region-based feature described in Sec. 3.2, will be mapped into the embedding space through a non-linear mapping function:

where Wv and bv are the parameters to be learned. σ() is an activation function. ˆ vt is the embedding feature of v∗ t .

Then, a recurrent layer of RNN is employed to seize the cooccurrence dependencies among attributes. Its hidden state ht is modeled as:

where mt-1 is the memory cell vector at time step t - 1 and wt is the embedding feature of the attribute at time step t.

Finally, an inference layer is built on top of the recurrent layer by considering the current hidden state ht and generating the distribution of the next attribute to be predicted via a softmax function:

where Wdh and bdh are the parameters to be learned. bdh is the bias term.

Attribute prediction. During prediction, we generate the attribute one by one under a greedy-decoding strategy according to the distribution φt at each time steps.

3.4 GenerationModule

The generation module aims to generate a sentence word by word, given the features of images. To leverage the attribute information, we denote the representation of attributes obtained from the inference module with a binary (0 or 1) attribute vector Is where 1 means that the image has the corresponding attribute, and 0 means not.

Since the attribute vector Is is rich in semantic information, we use it to initialize the hidden state of RNN as in [Liu et al., 2017]:

The idea behind this is that we force the RNN to understand the image in the beginning so that it could properly select or leave out some aspects about the image along its decoding process.

We make the generation module perceive the information of images by means of the attention mechanism. Specifically, at each time step t, a attribute-based feature vector, c∗t , is extracted in the attention module (see Sec. 3.2). c∗t will be further projected into the embedding space:

where Wc and bc are the parameters to be learned. σ() is an activation function. Then the RNN generates the next word by conditioning on the embedding feature ˆ ct and the representation of word yt:

where ¯ht-1 and ¯mt-1 are the hidden state vector and the memory cell vector of the decoder RNN at time step t-1, respectively. ¯ φt+1 is the distribution of the next word given the past information. yt is the embedding feature of the current word via mapping the one-hot representation of the word into the word embedding space.

In the proposed approach, the attribute vector Is is integrated into the decoder as a semantic regularisation, which is similar to [Liu et al., 2017]. But different from [Liu et al., 2017], the regularisation vector in our approach is represented as a hard-code way, where the element is either zero or one, instead of a soft-code way, where numbers ranging from zero to one depict the likelihood of the corresponding attribute to be related to the image. Besides, we also extract the context features for attributes and incorporate them into the captioning decoder by the attention mechanism.

Sentence generation. During generation, at time step t, we plug the word sampled from the distribution at last time step, ¯ φt-1, into the RNN. We employ the beam search strategy to boost the performance.

4 Experiment

4.1 Dataset and Setting

Dataset. Following previous works [Yao et al., 2017; Yang et al., 2016], we conduct experiments on the popular MS COCO dataset [Lin et al., 2014], which consists of 82783 training images and 40504 validation images. All images are labeled with at least 5 captions by human labellers. It provides 40775 images as a test set for online evaluation as well. For offline evaluation, we follow most previous works [Chen et al., 2017a; Yao et al., 2017] and split the 123287 images into three parts, 5000 for validation, 5000 for test and the remains for training.

Evaluation metrics. To compare with other methods, we use the same evaluation metrics, including BLEU [Papineni et al., 2002], ROUGE-L [Lin and Hovy, 2003] and CIDEr [Vedantam et al., 2015]. Meanwhile, we use the MS COCO caption evaluation tool 1 to compute the scores.

1 https://github.com/tylin/coco-caption

Compared approaches. We compare the proposed approach with the following state-of-the-art approaches. (1) NIC [Vinyals et al., 2015]: a standard neural network based approach which only injects the image into RNN at the initial time step. (2) HA and SA [Xu et al., 2015]: it incorporates the spatial feature map into the decoder via attention mechanisms. HA incorporates the feature map into the decoder by adopting a hard way, and SA adopts a soft way. (3) SCA-CNN [Chen et al., 2017a]: SCA-CNN adopts a spatial and channel-wise attention for captioning. (4) ATT [You et al., 2016]: ATT firstly predicts the attributes for images with CNN, and then adopts the semantic attention to adaptively select the attribute features in the decoder for captioning. (5)SCN-LSTM [Gan et al., 2017]: SCN-LSTM proposed a semantic compositional network by leveraging the attributes. (6)MSM [Yao et al., 2017]: MSM explores different ways to integrate the attribute information into the RNN.

4.2 Implementation Details

We convert all sentences to the lower case, and filter rare words that occur less than 5 times, and we end up with a vocabulary set of 9487 tokens. We use ResNet-101 [He et al., 2016] pre-trained on the ImageNet dataset to extract the image features. We do not crop or scale any image. Instead, we use the final convolutional layer of ResNet as image features, and apply spatially average pooling, resulting in a fixed size of 14 × 14 × 2048 of the feature map. The hidden state size of LSTM, the embedding dimension of the input word and the embedding dimension of image features are all fixed to 1000. During training, the parameters are updated by ADAM optimizer with a learning rate of 5 × 10-4 and 0.9 as the learning rate decay factor. We let the learning rate decay every 2 epoches and we train the model for 30 epoches with a batch size of 16. Following [Yao et al., 2017; Fang et al., 2015], we use the 1000 most frequent words in the training captions as the attribute vocabulary, which cover the majority of words in the training data. To train the inference module, for each image we rank the attributes according to their frequency. For caption generation in the testing stage, we apply the beam search algorithm to boost the performance by default. The beam size is empirically set to 3.

4.3 Evaluation

Compared with other approaches. The comparisons are shown in Tab. 1. The proposed model defeats NIC model with a large margin, which shows that the attribute information is effective and can provide helpful high-level information to enhance the caption generation performance. Compared with the visual attention based models, e.g., HA, SA and SCA-CNN, our model also has advantages in all metrics. Compared to SCN-LSTM, MSM and ATT models, which also adopt the attribute as high-level image features, the performance of our model is superior to theirs. We also train our model with reinforcement learning [Rennie et al., 2017], and submit the result of the official test set to the test server. The compared results are shown in Tab. 2. Our model can outperform other approaches in most metrics, even though we does not utilize the ensemble approach. We left the ensemble approach as a future work. The comparisons with the state-of-the-art approaches demonstrate the effectiveness and superiority of the proposed approach.

Ablation study. Ablation study is to verify the effectiveness of each component of our proposed approach. The results are shown in Tab. 3. First, to prove the effectiveness and superiority of the proposed approach inferring attributes via RNN, we adopt the same architecture of the captioning model as the best model in [Yao et al., 2017], and compare the performance with that reported in their paper. Our model can obtain a better performance than [Yao et al., 2017] (see the second row), which shows that the proposed approach to predict attributes is effective. Second, to reveal the effectiveness of the proposed attention module which provides the context features corresponding to attributes for the generation module, we train a captioning model with the standard visual attention mechanism directly attending to the feature map from CNN, indicated as "Attend to regions". And we compare it with our generation model without the semantic regularisation, indicated as "Attend to attributes" (see the third row). The result shows that the context features we propose for captioning model is effective and can potentially replace the original feature map from CNN. The performance improvement can be attributed to that the features are derived from the supervision of the semantic information and thus are more semantically powerful than features directly extracted from CNN. By incorporating the attributes as semantic regularisation and by attending to the context features derived from the inference module, the performance of our full model can be improved further (see the fourth row).

4.4 Qualitative Analysis

Fig. 3 presents a few examples generated by our model. Our inference module can detect various kinds of attributes successfully, including object terms, relational terms and descriptive terms. Since the true attributes used to train the inference model are collected from the ground truth captions, they may not completely describe the content of the image. But our inference model can still produce attributes outside the ground truth but highly related to the content of the image. For example, for the image in the second row, the attributes, traffic and lights, predicted by our inference module, are not in the ground truth, but highly related to the image. Moreover, these attributes can assist the captioning model to generate a descriptive sentence, as illustrated in the examples.

5 Conclusion

In this paper, we proposed an attribute-driven attention model for image captioning. We trained an attribute inference module by utilizing a CNN-RNN framework to model the co-occurrence dependencies among attributes. Different from other attribute-based approaches, we incorporate the attribute information and their corresponding context features into the decoder for sentence generation. The context features corresponding to attributes contain rich semantic information and thus are more compact for representing images than the feature map in the CNN. To verify the effectiveness of the proposed approach, we conducted experiments on MS COCO, a popular dataset for image captioning. We also compared the proposed model with other state-of-the-art captioning models. The results well demonstrated the effectiveness and superiority of the proposed approach.