Dependent Multi-Task Learning with Causal Intervention for Image Captioning

1 Introduction

Image captioning is a typical vision-to-language problem, which aims to automatically generate descriptions for images [Chen et al., 2015]. Recently, a line of studies used an extractthen-generate paradigm with a preliminary task, object detection [Anderson et al., 2018]. These studies first extracted a sequence of object-based region features from the images via a pre-trained detection network, and then used LSTMbased [Hochreiter and Schmidhuber, 1997] or Transformerbased [Vaswani et al., 2017] sequence-to-sequence (seq2seq) models to generate captions. Empirically, the pre-extracted features greatly help models produce high-fidelity captions [Anderson et al., 2018].

∗Corresponding authors

Although the paradigm achieved promising performance, we have observed two problems in the generated captions: 1) content inconsistency; 2) not informative enough. Regarding the first problem, given an image containing a "man" with long hair, we found that models often generated "woman" instead, which led to a contradicting fact 1 . Another example for the second problem is that, given an image with the subjectobject-predicate triple ("man", "playing", "Wii game"), we found models ignoring "Wii game" and using the quite general words "video game" instead. The generated caption would not be judged wrong but not informative enough.

From a causal perspective, the reason is that the models are inclined to build spurious connections between certain attributes of images and high-frequency categories or predicates. For instance, models are possible to capture the spurious statistical correlation between the visual features of "long hair" and "woman" through the confounder "female". Once "long hair" is detected, models may incorrectly predict "man with long hair" as "woman". To further explain this phenomenon, we show a typical causal graph in Figure 1. Let x and y be the given image and one ground-truth caption, respectively. We show the directed acyclic graph (DAG) including the mediator, the confounder, and the covariate. A robust model is supposed to utilize the mediator for prediction, which has causal effects on y, rather than utilizing the dashed lines for prediction.

1 The mentioned image and the tested model can be found in Section 4.4 and Section 4.1, respectively.

In this paper, we propose a dependent multi-task framework with the causal intervention (DMTCI) to eliminate the spurious correlations. Firstly, we formulate the image caption as a dependent multi-task (DMT) learning problem [Chen et al., 2020], which includes two tasks: bag-of-categories (BOC) generation and caption generation. The outputs of the first task are considered as the mediators, which are already annotated in certain datasets like MSCOCO and also utilized by recent work [Wang et al., 2020; Li et al., 2020]. Secondly, in terms of the confounders, we are probably to have no access to the real confounders. For example, "female" is not annotated. Fortunately, an available approach is using socalled "proxy confounders" [Louizos et al., 2017]. Since the spurious correlations problem we observe is that the models over-generating high-frequency categories or predicates, we let the proxy confounders be sampled from a set of concepts including high-frequency predicates and fine-grained object categories. Then we apply the variational inference [Kingma andWelling, 2014] to estimate the real confounders and let it sampled from a prior distribution. As a side advantage, our model can also infer covariates from the captions. Finally, we apply a multi-agent reinforcement learning strategy (MARL) to enable end-to-end training and reduce the inter-task error accumulations. An extensive set of experiments demonstrates that our model outperforms the baseline models and achieves competitive performance with those meticulously designed Transformers on the MSCOCO dataset. The main contributions of this work can be summarized as follows:

• We propose a dependent multi-task framework for image captioning, considering the BOC generation as an intermediate task to alleviate the inconsistency problem.
• We further introduce a latent de-confounding approach, which reduces the spurious correlations and estimates richer confounders through the variational inference.
• We cooperate with a multi-agent reinforcement learning approach to better utilize the task dependencies based on our multi-task framework and reduce the inter-task error accumulations.

2 RelatedWork

We categorize recent related work on image captioning into three groups according to the different focus of the work.

Architectural Innovation. The first category of work mainly focused on developing novel networks with powerful representation abilities. Some early studies encoded each image as a single feature vector through a pre-trained convolutional neural network (CNN), and then fed it to recurrent neural networks (RNNs) based on long short-term memory units [Vinyals et al., 2015; Donahue et al., 2015]. Afterward, Xu et al. [2015] explored the attention mechanisms for image captioning, which could dynamically focus on salient features at each decoding step. Another important line of work cooperated the task "object detection" for image captioning [Karpathy and Li, 2015; Fang et al., 2015]. Anderson et al. [2018] extracted object-level features by a pre-trained Faster R-CNN model, and combined them with the attention mechanism, which achieved significantly higher performance than objectfree models. Follow-up researches further captured the relationships among objects by adding self-attention on objectlevel features [Huang et al., 2019], or using graph convolutional networks [Yao et al., 2018; Yang et al., 2019]. More recently, Transformer-based [Vaswani et al., 2017] models were proposed with spatial position embeddings and better normalization layers [Guo et al., 2020], meshed attentions [Cornia et al., 2020] or higher-order attentions [Pan et al., 2020]. It is worthwhile to mention that almost all the state-ofthe-art (SOTA) models utilized the self-critical reinforcement (SCRL) strategy [Rennie et al., 2017].

Knowledge Enhancement. The second category of work involved prior knowledge. Kim et al. [2019] proposed a relationship-based captioning model that considered part-ofspeech (POS, i.e. subject-object-predicate categories) tags of captions, and therefore introduced beneficial inductive bias. Shi et al. further explored the semantics conveyed in the subject-object-predicate triples to enhance image caption generation [Shi et al., 2020]. The subject-object-predicate triples are extracted from captions by a textual scene graph parser introduced in [Schuster et al., 2015].

Causal Intervention. The third category of work is quite related to our work, which mainly focused on reducing the spurious correlations with causal intervention. Wang et al. used causal intervention to better learn the region features of the object detection task [Wang et al., 2020]. When trying to learn the conditional probability between two objects like "toilet" and "person", they assumed other objects as confounders and learned the objective of p(toilet|do(person)) by cutting off the link between other co-occurred objects and the object "person". Yang et al. assumed the dataset as the confounder and adjust it as concepts, such as "apple" and "green". Then they cut off the link between these concepts and image features.

3 Methodology

In this section, we first formulate the image captioning task as a dependent multi-task learning problem in Section 3.1. Then we illustrate our mediation and de-confounding approach in Section 3.2 and 3.3, with the model details in Section 3.4. Finally, we introduce the multi-agent reinforcement learning approach in Section 3.5.

3.1 Dependent Multi-Task Learning

Given the image and one corresponding ground-truth caption (x, y), we use a pre-trained Faster R-CNN network [Anderson et al., 2018] to extract a sequence of features R ∈ RLr×d from x where Lr and d denote the number of the detected objects and the dimension of features, respectively. The pretrained Faster R-CNN network is fixed during further training. Recent work mainly formulated image captioning as a single-task learning problem, directly maximizing p(y|R) while we formulate image captioning as a DMT problem, which maximizes the log-probability:

where m is an intermediate variable conveying important information to generate final captions. We choose the BOC as m which are already annotated in MSCOCO dataset and also utilized in recent work [Wang et al., 2020; Li et al., 2020].

3.2 Mediation with DMT

There is a train-test discrepancy in Equation 1. In the training stage, we can use ground-truth object categories to supervise the learning objective log p(m|R) while in the inference stage, m is not available. To reduce the train-test discrepancy, we include ˜m, the predicted BOC, for further training after pre-training with Equation 1. The learning objective of caption generation becomes:

where E denotes the expectation operation. Then the multitask objective is maximizing:

which utilizes the dependency between tasks and meanwhile reduces the train-test discrepancy. In the period of maximum likelihood estimation (MLE), we use the Gumbel-softmax approach [Jang et al., 2017] when sampling ˜m ∼ p(m|R) which can back-propagate gradients between tasks (details can be found in Appendix A in the longer version on Arxiv 2 ).

We view Equation 2 from the perspective of causal intervention. A typical causal graph including m as a mediator variable is shown in Figure 2. The approach of causal intervention relies on the do-calculus [Pearl, 2010]. Specifically, when applying the do(·) operation on the input variable X, we are trying to intervened X to the observed value x denoted by do(X = x). After the do-calculus, x (represented by R in practice) is not determined by other variables. In other words, all the arrows pointing to x will be cut off while the arrows from x to other variables remain. It means that, for the mediation causal graph, causal intervention will let p(y|do(R)) = p(y|R) which does not modify Equation 2.

2 https://arxiv.org/abs/2105.08573

3.3 Latent De-Confounding

However, for the de-confounding causal graph, the causal intervention will cut off the arrows pointing to x. We denote the proxy confounder and the real confounder by c and zz, respectively. We apply the do-calculus on x and obtain the deconfounding graph shown at the bottom of Figure 2. Firstly, we only consider c and obtain the probability distribution:

which makes p(y|do(R)) ̸= p(y|R) as p(c|do(R)) = p(c). It means that only considering the mediators m is not optimal during causal intervention. When jointly considering the mediator and the confounder, we do an intervention so that the mediator m is no longer affected by the confounder c. Then we have:

according to the mediation-confounding graph in Figure 2.

The next important thing is choosing the proxy confounder c and estimating the real confounder zc. Generally, confounders will have causal effects on x and y. For example, "female" may have causal effects on the visual features of "long hair" in x and the word "woman" in y. Those models modeling a high posterior probability q(female|long hair) may incorrectly take a man with long hair as a woman. In this case, "female" may be a workable confounder but not be annotated. Thanks to the so-called proxy confounder, we can estimate zc from proxy confounders c (e.g., "girl"). Since we have observed that models over-generated high-frequency expressions, we assume that the proxy confounder c can be sampled from a concept set that includes the high-frequency predicates and fine-grained object categories. We use the summarized predicates and fine-grained categories in [Shi et al., 2020] as the concept set. The real confounders zc has causal effects on c and we show the final causal graph of mediation and latent de-confounding in Figure 2.

Estimating zc requires the variational inference [Kingma and Welling, 2014], where the probability distribution p(zc)

is assumed to be priorly defined, such as a zero-mean unitvariance Gaussian distribution. Taking the modeling of the prior distribution p(c) for example, we can use following approximation:

where the integral is approximated by maximizing the evidence lower bound (ELBO), and ϑ and ϕ denote the parameters of two networks modeling the prior and the posterior distributions, respectively. And KL(·||·) denotes the KullbackLeibler divergence between two distributions. Then, we modify Equation 5 when considering zc as the confounder:

where ˜c ∼ p(c) means that we randomly sample concepts from the dataset and zc ∼ qϕ(zc|˜c) means that we can infer the actually mattered confounders in the latent space. It is worth noting that we are not designing a network to estimate the posterior distribution qϕ(zc|y, c) because y and c are not strongly coupled. Instead, as introduced in Equation 7, we model p(c) in the dataset rather than making a too strong assumption of c paired with y.

The final multi-task learning objective is maximizing:

where log pθ,ϕ(y|do(R)) and log pϑ,ϕ(c) are computed through Equation 7 and 6 respectively.

3.4 Model Details

To implement the final learning objective, we introduce our model in detail, which can be divided into three main parts: the hierarchical encoder, the BOC generator, and the caption generator.

The Hierarchical Encoder

Given the sequence of object features R, we use a hierarchical encoder for further encoding, which consists of multiple layers of multi-head attention network followed with "adding and normalization" layers:

where i denotes the layer number, R(0) = R, and Attn(·) is the multi-head attention network introduced in [Vaswani et al., 2017]. We choose the outputs of an intermediate layer R(im) to predict the BOC m, and the outputs of the final layer R(n) to predict the caption y.

The BOC Generator

The first task is a category-level classification task. Fo each image, we convert m to the one-hot label matrix Y m ∈ RLm×Nm where Lm denotes the number of all categories, and Nm denotes the maximum number of detected objects belongs to each category. We use another attention-based network for prediction:

where Qc ∈ RLm×d denotes the category-specific query matrix in the attention layer and is trainable. Then we use a fullyconnected neural network (FCNN) followed with the softmax function along the last dimension to generate the probabilities of object numbers ˜ Y m. Then the loss of the first task is computed through cross-entropy:

where Y mi and ˜ ymi are two one-hot vectors representing the number of detected objects belong to the j-th category in Y m and ˜ Y m, respectively. Maximizing the log probability log pφ(m|R) is implemented by minimizing Lm.

The Caption Generator

In the second task, we mainly introduce how to implement pθ(y|R, ˜m, zc, ˜c) and qϕ(zc|c).

Firstly, we need to re-encode the predicted labels ˜m. Taking the gold label m for example, we involve another two sequences wm = {wm j } and idm = {idmj } representing the words and category indexes, respectively. For example, wm = [person, dining table, pizza, ...] and idm = [1, 2, 3, ...], and m marks the number of corresponding categories. We encode the words of each category in wm with a LSTM layer:hmj,t = LSTM(OH(wm j,tW w), hmj,t-1 ) (12)

where OH(·) denotes the one-hot operation, wm j,t denotes the t-th token in the j-th category in wm, and Ww denotes the word embedding layer. Assuming Hm = {hmj,Tm } containing the hidden states at final tokens (at the Tm-th step) and mj representing the occurrence numbers of j-th category in the BOCm, we fuse three kinds of embeddings as follows:

where ⊙ denotes the element-wise multiplication and σ denotes the sigmoid function. Wo,Wm denote parameters of the category embedding and number embedding layers, respectively. In this way, the final embedding matrix Em = {emj } ∈ RLm×d is category-words-aware and number-aware. Then we use the max-pooling operation to obtain a vector zm representing the global information for decoding. It is worth noting that we do not further encode Em with position-aware encoders such as LSTM or attention networks, because the BOC does not contain the detailed information. For example, BOC can not indicate whether a person is hitting a ball or holding a ball. Therefore, we only apply the pooling operation and get zm.

Secondly, we need to infer zc through the network qϕ(zc|c). We extract the high-frequency predicates and finegrained categories from y and convert them to an unordered set c. We embed c to Ec ∈ RLc×d through the word embedding layer and pass it to a two-layer Transformer without the position embedding, and obtain a sequence of hidden states Hz ∈ RLc×d . Since qϕ(zc|c) is often approximated to a Gaussian distribution, we use the mean pooling of Hz followed with two FCNNs to get the corresponding mean and variance vectors, and then zc is sampled from this Gaussian distribution.

Thirdly, we need to implement pθ(y|R,˜m, zc, ˜c) where ˜m ∼ pφ(m|R), zc ∼ qϕ(zc|˜c) and ˜c ∼ p(c). After randomly selecting ˜c from the dataset, we obtain a vector zc by sampling from qϕ(zc|˜c). Since ˜m has been already encoded to zm, we simply use a top-down attention LSTM network [Anderson et al., 2018] to decode each token:

where yt denote the t-th token of y, and y<t denotes the sequence of tokens preceding yt. The latent variable zm and zc are concatenated with the hidden states at all steps. Note that our model is also compatible with Transformerbased decoders. But recent empirical work has shown that Transformer-based decoders require meticulous designs, such as meshed attention [Cornia et al., 2020] or high-order attention [Pan et al., 2020], and greatly increase the time complexity when optimized with reinforcement learning (RL).

3.5 Multi-Agent Reinforcement Learning

RL is commonly used in the task of image captioning to reduce the train-test discrepancy [Ranzato et al., 2016]. Since we formulate the image captioning as the dependent multitask problem, the inter-task train-test discrepancy also needs to be tackled. During the MLE optimization period, Gumbelsoftmax approximation is used but will improve the gradient variance because of the temperature parameter [Nie et al., 2019]. Therefore, we use a multi-agent reinforcement learning strategy to train the model end-to-end based on the SCRL strategy on the two tasks alternatively.

SCRL for the BOC Generator. The Monte-Carlo sampling is used to generate ˜m and the arg-max operation to generate the baseline m. The gradient of the BOC generator can be expressed by:

where r1(˜m) is the reward provided by both two tasks. After feeding ˜m to the caption generator, we obtain y˜m through greedy decoding and compute the reward r1(˜m) by:

where λ1 and λ2 are the hyperparameters to balance the two rewards, and s1(·) denotes the score function to match two BOCs, and s2(·) denotes the CIDEr-D score function [Vedantam et al., 2015].

SCRL for the Caption Generator. Fed with the baseline BOC m and a sampled latent confounder zc ∼ p(zc), the caption generator can obtain ˜ y by the Monte-Carlo sampling and y by the greedy decoding. We use the SCRL strategy with the gradients computed as follows:

where r2(˜ y) = s2(˜ y, y). Furthermore, we adopt the maxmax framework [Xiao et al., 2020] which maximizes the rewards of the two agents alternatively.

4 Experiments

4.1 Datasets and Settings

Datasets. We experiment on the MSCOCO 3 dataset, which is the most popular dataset for image captioning. The original dataset contains about 82,783 training images and 40,504 validation images. Following most of the previous work, we first evaluate our model on the Karpathy data split [Karpathy and Li, 2015] with 5,000 images for validation, 5,000 images for testing, and the rest for training. Then we evaluate our model on the on-line COCO test server. We set the maximum decoding length to 16 which equals the maximum length of 95% of captions. We keep the words that occur no less than 5 times, resulting in a vocabulary of 10,369 words.

Evaluation and Settings. We apply five standard metrics for evaluation: CIDEr-D [Vedantam et al., 2015], BLEU [Papineni et al., 2020], METROT [Banerjee and Lavie, 2005], ROUGE [Lin, 2004], and SPICE [Anderson et al., 2016]. All the metrics are computed with the publicly released code. 4

Compared Models. We compare our models with several models including SCST [Rennie et al., 2017], Up-Down [Anderson et al., 2018], AoANet [Huang et al., 2019], UpDown+VC [Wang et al., 2020], WeakVRD [Shi et al., 2020], M2 Transformer [Cornia et al., 2020], and XLAN [Pan et al., 2020]. Among them, XLAN is the recent SOTA with an advanced architecture, and Up-Down+VC is related to our work which uses causal intervention to reduce spurious correlations between objects. Also, we compare with the baseline model, TransLSTM, consisting of a Transformer encoder and an updown LSTM attention decoder. It is worth noting that one may find that a recent model OSCAR [Li et al., 2020] greatly outperforms the models mentioned above, but we want to clarify that OSCAR made two unfair settings: 1) using more vision-language datasets for pre-training, including Conceptual Captions, SBU captions, Flicker30k, and GQA [Li et al., 2020]; 2) utilizing the ground-truth rather than generated object categories in the inference phase.

Hyperparameters. We use the pre-trained Faster R-CNN model [Anderson et al., 2018] to represent each image as an adaptive sequence of feature vectors R where the number of detected objects n ∈ [10, 100]. The number of encoder layers is 4 and we use the output of the second layer to predict BOC. The dimension of each feature vector vi is 2,048 and then projected to 1,024. The dimensions of the word embedding layer, attention layers, and the LSTM-based decoder are all set to 1,024. The learning rate is initialized to 0.0001 and decreased by half when the CIDEr-D score does not increase in 2 epochs, with the minimum learning rate set to 5 · 10-6 . The batch size is set to 50. The model is firstly optimized with MLE for 30 epochs (Gumbel sampling ˜m after 15 epochs), and then optimized with MARL for another 35 epochs. The special setting of our model is that we can sample different zc to generate multiple captions. Therefore, we train an imagetext selector to select the best of 20 generated captions for each image (details in Appendix B in the longer version on Arxiv).

3 http://cocodataset.org/

4 https://github.com/tylin/coco-caption

4.2 Main Results

Table 1 presents the performance on the Karpathy test split of our model as well as the compared models. As shown, our model DMTCI achieves average CIDEr-D scores of 120.2 and 131.7 and outperforms the baseline TransLSTM by 2.9 and 3.0 points in MLE and RL periods, respectively. When compared with the related model with causal intervention, "Up-Down+VC", our model outperforms it by 1.2 points on the CIDEr-D score. Furthermore, when combining with the trained selector, our model outperforms TransLSTM by 6.4

and 3.3 points in MLE and RL periods, respectively. And our model achieves competitive results with the recent SOTA model XLAN, without using meticulously designed attention mechanisms. It is worth noting that we also reproduce XLAN with the officially released codes, because in recent work, a slightly tricky method is to obtain an ensemble model by training the model multiple times. Thus, the single-model performance is reported under the best trial.

If we have the gold BOC, the averaged CIDEr-D score can be further improved to 122.9 and 135.2 in MLE and RL periods respectively. It suggests that future work can focus on improving the performance of the BOC generator. Moreover, if we have an optimal selector, we can further improve the CIDEr-D score to 141.9 in the MLE period even without RL, which can be considered as the upper bound of our model. And RL slightly decreases the upper bound, which means that optimizing with RL drops some useful patterns.

Table 2 presents the single-model results of the models on the online COCO test server. Compared with recent SOTA models which have reported single-model performance, DMTCI achieves competitive performance and achieves the best CIDEr-D score with 40 reference captions.

4.3 Ablation Study

To analyze which mechanisms are driving the improvements, we present an ablation study in Table 3. We obtain ablated models including: 1) Pipeline, learning the two tasks sequentially; 2) DMTCI-M, only considering the mediator; 3) DMTCI-LDC, only considering the confounder. To avoid misleading, we denote our full model by DMTCI-Full. Table 3 shows the performance of these ablated models. We can see that Pipeline suffers from the train-test discrepancy and falls behind the baseline model TransLSTM. DMTCI-M increases the CIDEr-D score by 2.2 and 2.6 points over TransLSTM in MLE and RL periods, respectively. DMTCI-LDC outperforms TransLSTM while performing slightly worse than the DMTCI-M. When compared with the two ablated models, our full model achieves the best CIDEr-D score in both MLE and RL periods. And the selector further brings a large improvement in the MLE period and does not help much in the RL period.

4.4 Case Study

To help understand how the mediators and the confounders affect the learning, we show some cases in Figure 3. The inconsistent descriptions are marked by the red underlines and the not informative enough descriptions are marked by the gray background. The bold blue words represent applaudable expressions that are content consistent and informative. Compared with TransLSTM, DMTCI-M can alleviate the inconsistency problem by involving the mediator. When further combining with the latent de-confounding approach, DMTCI-Full generates consistent and informative captions.

Furthermore, we are trying to discuss why involving the mediators and the confounders work. At present, deep learning models have strong abilities to capture correlations. However, since it is too strong, models can often utilize the cooccurrence of concepts to minimize the training loss, ignoring the exact appearance of the detected objects. Then in the inference stage, it is possible to infer non-exist objects relying on the co-occurrence (e.g., "brush" and "woman"). Involving mediators and confounders during training could alleviate this problem by focusing on the exactly appeared objects.

5 Conclusion

In this paper, we propose a dependent multi-task framework with causal intervention for image captioning. We first involve BOC as the mediator to help the model understand the pre-extracted features, then we propose a latent deconfounding approach to alleviate the spurious correlation problem. Furthermore, we apply a multi-agent reinforcement learning strategy to better utilize the task dependencies and enable end-to-end training. The experiments show that our model yields competitive results with recent SOTA models and the proposed framework could improve content consistency and informativeness.

Acknowledgments

The authors would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China under Grant 2018YFC0830400, and Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0102.