A Multi-task Learning Approach for Image Captioning

1 Introduction

Humans have the remarkable capability to describe an image verbally because they are naturally multi-task intelligent agents. Humans have developed those skills since childhood by not only learning to perform a single task, but rather adapting to comprehend the complex outer world via multi-channel perceptions and communications. They are trained in fact by performing multiple relevant tasks together to develop a comprehensive set of skills to understand and describe a scenario. If one desires to create a machine intelligence imitating such a comprehensive skill set of human, studying all relevant tasks that contribute its development is quite a necessary step.

In this paper, we are motivated by the fact that a cognitive AI is by nature multi-tasking and develop a computerized image captioning agent that can also perform a few other related tasks. Image captioning, as to generate a sentence describing the salient aspects of an image, is a fundamental task in computer vision and natural language processing [Bernardi et al.,

∗Min Yang is corresponding author (min.yang@siat.ac.cn)

1 Codes are publicly available at https://goo.gl/iZtCBB.

w/o syntax: a man holding a tennis ball with a tennis ball.

w/o object: a man holding a tennis ball.

Our Model: a man hitting a tennis ball with a tennis racket.

a suitcase filled with lots of items and. a suitcase filled with lots of items. a suitcase filled with lots of items on a bed.

Table 1: Examples of captions that fail to respect to syntax structure and missing salient object "bed" and "racket".

2016]. In recent years, it is often approached with supervised learning framework by collecting human generated examples and developing models that base on matching the generated text to those collected annotations. We suppose such learning frameworks are theoretically of limited success in two dimensions. First, the model created from the collected data can only comprehend the complexity of the problem subject to the degree of complexities of presented examples. Since the dataset is more or less a finite collection, the presented complexities should be limited. Second, the loss function used in numerically optimizing the model is often not sensitive to certain aspects of the structured output that have not been emphasized in the conventional evaluation metric of image captioning, such as object categories and syntax of generated sentences. In fact, we show that exposing the learning framework with more relevant data and objectives can be helpful in both dimensions.

Multi-task learning is hardly a new idea for machine learning, but often remains as a non-trivial step for building empirically successful systems. We argue that it is essentially the case for creating an effective image captioning system. In an ablation study of our models (see Table 1), a model which is unaware of the sentence syntax could generate a broken sentence for describing a picture; a model which does not recognize all presented objects could generate a descriptive sentence missing a salient object in the picture. Yet, an improved system whose components have been co-trained with multiple related tasks could generate a more satisfied sentence for accurately and correctly describing a picture. Based on this observation, we believe a multi-task learning framework helps to improve an image captioning system in dimensions that have not been quantitatively measured in conventional evaluation metrics.

Our system exploits the recent successes of the encoderdecoder framework to generate image captions [Karpathy and Fei-Fei, 2015; Vinyals et al., 2015]. The common idea of this framework is to use a convolutional neural network (CNN) as an encoder to extract features representing the visual understandings of an input image and then feed the feature vector to a recurrent neural network (RNN) based decoder so as to generate image captions. Sharing this standard framework with other related approaches, we in this paper propose additional regularizations using multi-task learning. Firstly, our CNN encoder is regularized with the co-training to perform an additional task of multi-object classification. Secondly, our RNN decoder is also regularized with the co-training to perform another task of syntax annotation [Nadejde et al., 2017]. The purpose of co-training is not to achieve the best performance on these auxiliary tasks, but rather to compensate for the missing regularization requirement of image captions in the standard framework.

We summarize our main contributions as follows:

• We propose MLAIC, a multi-task learning system to jointly train the task of image captioning and two other related tasks: multi-object classification and syntax generation. The auxiliary tasks help to enhance the CNN encoder and the RNN decoder in the image captioning model. Specifically,
• Multi-object classification co-trained with image captioning intends to learn an object-rich image encoder and improves the quality of locating contextual information of an image.
• The variations of style and wording of captions with respect to different object categories are explored under controlled experimental settings.
• The RNN decoder is capable to leverage word-level syntax to generate high-quality captions from language modeling perspective. It alleviates the issues of incomplete sentences and duplicated words.
• The experimental results show that MLAIC achieves outstanding performance on the widely used MSCOCO dataset according to both the offline Karpathy test spilt and the online server evaluation.

2 RelatedWork

Generating text descriptions from images is a challenging problem. Bernardi et al. [2016] provided a detailed review of most existing approaches, the benchmark datasets, and the evaluation measures for image captioning.

Recent advances in deep neural networks have substantially improved the performance of image captioning task. A typical image captioning strategy is to combine CNN and RNN [Karpathy and Fei-Fei, 2015; Vinyals et al., 2015], where CNN is used to extract the compact representational vector of a whole image, and RNN is used to construct the language model operated on the representation vectors to generate captions. Visual attention has been proven as an effective way to improve the basic encoder-decoder framework. For example, Xu et al. [2015] introduced an attention-based model that automatically learn where to attend when generating image descriptions. The attention is modeled as spatial probabilities that re-weight the feature map of the last convolutional layer in the CNN. Chen et al. [2017] dynamically modulated the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention was.

There have been increasing interests in integrating the encoder-decoder framework and reinforcement learning paradigms for image captioning [Liu et al., 2016; Rennie et al., 2016; Zhao et al., 2017]. For example, Liu et al. [2016] employed policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr metrics, where the SPICE score ensured the captions were semantically faithful to the image, and CIDEr score ensured the captions are syntactically fluent. Rennie et al. [2016] proposed a self-critical sequence training (SCST) method by employing the popular REINFORCE algorithm.

Multi-task learning is a useful learning paradigm to improve the supervision and the generalization performance of a task by jointly training it with related tasks [Caruana, 1998]. Venugopalan et al. [2016] explored linguistic improvements to the video caption decoder by fusing it with external language models. Pasunuru and Bansal [2017] improved video captioning by sharing knowledge with two related directedgeneration tasks: a temporally-directed unsupervised video prediction task and a logically-directed language entailment generation task.

Our model differs from the above approaches. We perform image captioning by multi-task learning, which shares knowledge with three related tasks: multi-label classification, image captioning, syntax generation, so that we can improve the performance of both the CNN encoder and LSTM decoder.

3 Our Model

Given an image x, for the object classification, we have yo = {yo 1 , yo 2 , ..., yoC } denoting the object vector of each image, where yo i = 1 if object i is annotated in this image; otherwise yo i = 0, C is the number of object categories. For image captioning, we have yw = {yw 1 , yw 2 , ..., yw T } denoting the image description given image x, where T is the length of sequence. For syntax generation, we have ys = {ys 1 , ys 2 , ..., ysT } denoting the Combinatory Category Gramma (CCG) supertag sequence with respect to the corresponding caption of image x. We use Ww, Ws and Wo to denote the vocabularies of captions, annotated CCG supertags and object categories, respectively.

Our model MLAIC, whose framework is illustrated in Figure 1, jointly trains the image captioning task with two related tasks: the object classification and the syntax generation. The object classifier shares its CNN encoder with the encoder of image captioning task. All object labels are encoded into lowdimensional distributed embeddings and treated as extra input to LSTM decoder, helping the LSTM decoder focus on different aspects of the images with respect to the object labels. The syntax generation task shares its LSTM decoder with the decoder of image captioning, which predicts words and syntax by training the shared LSTM decoder in a similar fashion as is done in [Nadejde et al., 2017]. Next, we elaborate the three tasks in details.

3.1 Shared CNN Encoder

We use the ResNet-101 [He et al., 2016] pre-trained on ImageNet [Russakovsky et al., 2015] as our CNN encoder to encode the input image x into L vectors, each of which is a D-dimensional vector corresponding to the features extracted at different locations of the image. These vectors are referred to as annotation vectors.

We also introduce zfc to represent object probability of image x calculated by fully connected layer: zfc = FC(z). Following the work of [Xu et al., 2015], we extract feature vectors on the lower convolutional layer rather than use the fully connected layer. In this way, the decoder can selectively attend to certain parts of an image by weighting a subset of the feature vectors. In our experiments, we use the 14 × 14 × 2048 feature map in ResNet-101. That said, our LSTM decoder operates on the flattened 196×2048 (i.e. L×D) representations. The shared CNN encoder is then fine-tuned with both image captioning and multi-object classification tasks.

3.2 Shared LSTMDecoder

The image captioning task shares its LSTM decoder with the decoder of syntax generation task. Following [Anderson et al., 2017], the decoder consists of two stacked LSTM networks. The first LSTM layer (denoted as LSTM(1) ) is characterized as a top-down visual attention model, while the second LSTM layer (denoted as LSTM(2) ) is a language model. The attention model takes input as the concatenation of the previous output of the language LSTM (i.e., h(2) t-1 ), the meanpooled image feature ¯ z = 1L ∑i zi, the word embedding of the previously generated word (i.e., ewt-1 ), and its syntax embedding (i.e., est-1 ):

where x(1) t is the input of LSTM(1) at time step t. The hidden state of attention LSTM at time step t is then computed by:

Given the output of the attention LSTM h(1) = {h(1) 1 , h(1) 2 , . . . , h(1) T }, where T is the length of the sequence, we compute the attended image feature ˆ z, which is then used as the input to the language LSTM. The attended image feature ˆ z makes sure that every time step of the decoder can get full information of the context. We calculate ˆ zt when we decode the t-th word by

Here, the attention weight αt,i for the i-th annotation of CNN encoder is computed by

where σ is a feed-forward neural network, which maps a vector to a real-valued score. This attention weights αt,i models the alignment between the image content at location i and the output word at position t.

We use the LSTM language model to produce a caption (or a CCG supertag sequence) by generating one word (or CCG supertag) at every time step, which takes as input the concatenation of the output of the attention LSTM (h(1) t ) and the attended image feature ( ˆ zt), given by:

where x(2) t is the input of LSTM(2) .

The hidden state of the language model LSTM at time t is then computed by

Finally, the generation probabilities of the t-th word and the t-th CCG supertag are given by

where U w , U s , bw and bs are the parameters to be learned.

Incorporating Object Label Embeddings To explore the variations of caption styles for images of different objects, we integrate the object label embeddings in the LSTM decoder. A forgetting gate is introduced to determine when the object label information contributes, which is computed by ft = sigmoid (h(2) t-1 , zfc). Then the biased word generation probability is given as:

where W is transformation matrix to project object label into word embedding space. IWo (·) is the indicator vector denoting whether candidate word w is in the vocabulary Wo of object categories.

3.3 Multi-task Learning

Our MLAIC model consists of three subtasks, each has its own training objective. For the object classification subtask, the model minimizes the multi-label margin loss function following the work of [Li et al., 2017]:

where θ is the set of parameters, zfc j denotes the probability that image x contains object j, wj is the weight indicating the frequency of occurrence of object j in the image, b is a hyper-parameter that determines the margin, commonly set to 1.0.

For the syntax generation and image captioning subtasks, we employ the minimum negative log-likelihood estimation:

For the purpose of improving shared CNN encoder and LSTM decoder, we train these three related tasks simultaneously. The joint multi-task objective function is minimized by:

For comparison with recent work [Rennie et al., 2016], we also optimize directly for CIDEr [Vedantam et al., 2015] using policy gradient algorithm, and minimize the negative expected rewards:J word

where r(.) is CIDEr score function. According to the policy gradient theorem [Williams, 1992], we compute the gradient of the expected reward with respect to parameters as:

After pre-training the proposed model by minimizing the negative log-likelihood with multi-task using Eq.13, we switch the model to further minimize a mixed training objective, integrating the reinforcement learning objective Jrl with the original multi-task loss.

where β is a hyper-parameter, and we set β = 0.005.

4 Experimental Setup

4.1 Dataset

In our experiment, we use the widely used MSCOCO 2014 image captions [Karpathy and Fei-Fei, 2015] as our dataset, which consists of totally 82,783 training, 40,504 validation, and 40,775 testing images, each of which has 5 ground truth captions. For the off-line testing, we adopt the commonly used Karpathy split [Karpathy and Fei-Fei, 2015], which uses 113,287 images for training, and 5,000 images for validation and testing, respectively. For the on-line server evaluation, our model is trained on 118,287 images and validated on 5,000 images.

For multi-label object classification, we use the MSCOCO detection dataset [Lin et al., 2014], which consists of 500,000 object instances from 80 different object categories. It contains the same images and shares the same data split with MSCOCO 2014 image captions dataset [Karpathy and FeiFei, 2015].

For syntax generation, following the work of [Nadejde et al., 2017], the captions of training data is annotated with CCG supertags by using EasySRL 2 , where each word has a corresponding dependency label of supertags.

4.2 Baseline Methods

In the experiments, we compare our model with stateof-the-art methods, and several recent strong competitors are included below: Adaptive [Lu et al., 2016], SCST: Att2in/Att2all [Rennie et al., 2016], TopDown: ResNet/UpDown [Anderson et al., 2017], StackCap [Gu et al., 2017a]. In the experiments, we use the same parameter settings as in the original papers.

4.3 Implementation Details

We first pre-train our model on the training data with crossentropy cost, and use Adam optimizer with an initial learning rate 5× 10-4 and a momentum parameter of 0.9 to optimize the parameters. After that, we run the proposed RL-based approach on the just trained model, which is directly optimized for the CIDEr metric. During this stage, we use Adam optimizer with learning rate 5×10-5 . We set λ1=0.2, λ2=0.7, λ3=0.1. We set the number of hidden units in TopDown attention LSTM (LSTM(1) ) to 1,000, the number of hidden units in language model LSTM (LSTM(2) ) to 512, the size of the input word embedding to 512, and the size of the CCG supertag embedding to 100. During the decoding stage, we use a beam size of 5 to generate captions.

2 https://github.com/uwnlp/EasySRL

4.4 EvaluationMetrics

To quantitatively evaluate our image captioning method, we follow previous work to use BLEU-N (N=1,2,3,4), METEOR, ROUGE, CIDEr scores for comparison. All these metrics measure the consistency between n-gram occurrences in generated captions and ground-truth captions, where this consistency is weighted by n-gram saliency and rarity.

5 Experimental Result

In this section, we compare our model with the competitors quantitatively and qualitatively.

5.1 Quantitative Evaluation

Offline Evaluation The experimental results onMSCOCO with Karpathy test split are summarized in Table 2. We observe that MLAIC substantially and consistently outperforms the existing methods by a noticeable margin on most of the evaluation metrics. In particular, our model successfully yields better scores of all evaluation metrics compared to the TopDown model which utilizes the same basic CNN-LSTM backbone as ours. Our model benefits from the fact that both the CNN encoder and the LSTM decoder are improved by jointly learning the image captioning with the object classification and the syntax generation tasks.

Online Evaluation Table 3 reports the performance of our single model on the official MSCOCO evaluation server 3 . The previous top performers on the leaderboard (as of December 7, 2017) are also demonstrated. Our model achieves better or competitive performance compared to the state-ofthe-art competitors (both single and ensemble models).

Ablation Study For the purpose of analyzing the effectiveness of different components of our model for image captioning, in this section, we report the ablation test of our model by discarding the multi-label object classification task (w/o object) or the syntax generation task (w/o syntax), respectively.

3 https://competitions.codalab.org/competitions/3221

In order to investigate the performance of the policy gradient update, we also report the results of our multi-task learning model with policy gradient update (i.e., CIDEr-optimization) or with cross-entropy optimization, respectively.

The results are demonstrated in Table 4. Generally, both tasks contribute, and the multi-label object classification task contributes most. This is within our expectation since the object classification task helps the CNN encoder learn better image representations. In addition, the proposed model utilizes the classification results (label embeddings) to produce captions of different styles and wording with respect to different object categories. While the improvement of integrating syntax generation is relatively limited. This may be because that the issue of the incomplete sentence has little effect on the evaluation metrics of image captions. As shown in Table 5, the sentences generated by w/o syntax model are incomplete but achieve high scores in terms of the evaluation metrics. It is no surprise that combining both tasks achieves the best performance for all evaluation metrics. The main advantage comes from its capability of sharing knowledge of image captioning with two related tasks and learning better encoder and decoder representations simultaneously. We can also observe that the model with policy gradient update substantially outperforms the model with cross-entropy by a noticeable margin on all the evaluation metrics. The is because that the policy gradient update is able to bypass the exposure bias and non-differentiable evaluation metrics issue, and maximize long-term reward in caption generation.

5.2 Qualitative Evaluation

To evaluate the proposed model qualitatively, we show some generated image descriptions in Table 5. It is easy to see that MLAIC can generate reasonably relevant and plausible sentences. For example, the sentences "a man hitting a tennis ball with a tennis racket" and "a bunch of cars and buses on a highway" created by MLAIC are more precise in describing the content of the images. By comparing the results of w/o syntax and MLAIC, we reveal that the syntax generation task is able to help the LSTM decoder to alleviate the issues of generating duplicate words and incomplete sentences. On the other hand, the object classification task makes the CNN encoder be able to recognize the existence of objects, and help the LSTM decoder produce captions of different styles and generate novel words with respect to different object categories.

The performance of the syntax generation task is also improved by the multi-task learning mechanism. We provide some qualitative examples of image captions and the corresponding CCG supertags in Table 6 to evaluate the syntax generation model. From Table 6 we observe that the generated CCG supertag sequences are reasonable and consistent with the corresponding images captions. For example, the supertag (S[dcl]\NP)/(S[ng]\NP) for the verb "is" in the first caption implicates that a present participle is expected if the next word is verb. While the supertag (S\NP)\(S\NP)/NP for "on" implies that a noun phrase will be generated in the next. Our model can predict the high-quality CCG supertags, and help the image captioning alleviate the problem of generating duplicate words and incomplete sentence.

6 Conclusion

We proposed a novel multi-task learning method to improve image captioning by jointly training the image captioning with two related auxiliary tasks: object classification and syntax generation. The object classification helped learn better image representations and improved the performance of visual attention, and the syntax generation helped alleviate the problem of generating incomplete sentences and duplicate words. To verify the effectiveness of our approach, we conducted extensive experiments on widely used MSCOCO dataset. The experimental results demonstrated that our model achieved impressive results compared to other strong competitors.