Learning to Discretely Compose Reasoning Module Networks for Video Captioning

1 Introduction

Video captioning, the task aims to automatically generate natural language descriptions for videos, has received increasing attention in computer vision and machine learning. Even though our community achieves the significant advance in visual recognition [He et al., 2016; Ren et al., 2016] and natural language understanding [Bahdanau et al., 2015], video captioning is still a very challenging task and far away from satisfactory for it not only requires a thorough understanding of the input videos, but also requires step-by-step visual reasoning along the generation process.

∗Equal Contribution

†Corresponding Author

As illustrated in Figure 1, to describe the video, we human may have the following reasoning process: 1) identifying the subject to describe of the video, i.e., "man", 2) inferring what is the man doing, i.e., "shooting", 3) identifying what is the object of "shooting", i.e., "basketball", 4) generating the final description "a man is shooting a basketball" by inserting several function words. In a nutshell, there are three fundamental reasoning mechanisms involved: 1) Locate one region to generate visual words, 2) Relate pairwise regions to generate action words, 3) Generate function words according to the language context.

Most existing video captioning methods [Venugopalan et al., 2015; Donahue et al., 2015] follow the encoder-decoder framework, where a CNN is employed as an encoder to produce the video features and an RNN is employed as a decoder to generate the captions. Those methods usually neglect the nature of the above human-level reasoning, thus hurting the explainability of the generation process. Even though there are some recent works have explored the visual reasoning in visual question answering [Andreas et al., 2016; Hu et al., 2017; Yang et al., 2019a] and visual grounding [Cirik et al., 2018; Liu et al., 2019; Hong et al., 2019] by decomposing the questions or referring expressions into a linear or tree reasoning structure with several neural modules, the situation in the video captioning is more challenging because 1) unlike still images, videos contain richer visual content thus requiring more complex visual reasoning over both space and time, 2) unlike questions or referring expressions which are given in advance, the video descriptions are not available during the inference. Therefore, the model must dynamically compose the reasoning structure along the generation process.

To tackle the above two challenges, we propose a novel video captioning framework named Reasoning Module Networks (RMN). Firstly, to perform visual reasoning over both space and time, our RMN employs three fundamental spatiotemporal reasoning modules: a) LOCATE module to locate one single region over the video by a spatial-temporal attention, thus generating the visual words, e.g., "man" and "basketball" in Figure 1; b) RELATE module to relate pairwise regions over the video by first detecting the object of each frame, and then modeling the action by pairing two frames, thus generating the action words, e.g., "shooting" in Figure 1; and c) FUNC module to generate the function words according to the language context, e.g., "a" and "is" in Figure 1. Secondly, to compose the reasoning structure along the generation process, our RMN employs a dynamic and discrete module selector. To jointly train both the three modules and the selector in an end-to-end manner, we adopt the recently proposed Gumbel approximation [Jang et al., 2017] to make the discrete sampling process differentiable, and then constraint the module selector with a linguistic loss of part-ofspeech (POS) tag labels.

We validate the effectiveness of the proposed RMN by conducting extensive experiments on two widely-used datasets MSVD [Chen and Dolan, 2011] and MSR-VTT [Xu et al., 2016]. RMN outperforms the state-of-the-art video captioning methods on most metrics. Qualitative results indicate the generation process is explicit and explainable.

Our main contributions are three-fold: 1) We propose a novel framework named reasoning module networks (RMN) for video captioning with three spatio-temporal visual reasoning modules; 2) We adopt a discrete module selector to dynamically compose the reasoning process with modules; 3) Our RMN achieves new state-of-the-art performance with an explicit and explainable generation process.

2 RelatedWork

2.1 Video Captioning

There are two main directions to solve the video captioning problem. In the early stage, template-based methods [Kojima et al., 2002; Guadarrama et al., 2013], which first define a sentence template with grammar rules and then aligned subject, verb and object of the sentence template with video content, were widely studied. Those methods are hard to generate flexible language due to the fixed syntactic structure of the predefined template. Benefit from the rapid development of deep neural networks, the sequence learning methods [Venugopalan et al., 2015; Yao et al., 2015; Pan et al., 2017] are widely used to describe the video with flexible natural language, most of these methods are based on the encoder-decoder framework. [Venugopalan et al., 2015] proposed S2VT model which regards the video captioning task as a machine translation task. [Yao et al., 2015] introduced a temporal attention mechanism to assign weights to the features of each frame and then fused them based on the attention weights. [Li et al., 2017; Chen and Jiang, 2019] further applied spatial attention mechanisms on each frame.

Recently, [Wang et al., 2019] and [Hou et al., 2019] proposed to leverage Part-of-Speech (POS) tags to boost video captioning. [Wang et al., 2019] encodes the predicted POS sequences into hidden features, which further guides the generation process. [Hou et al., 2019] mixes word probabilities of multiple components at each timestep conditioned on the inferred POS tags. However, both of them lack the reasoning capability for rich video content. On the contrary, we propose three well-designed reasoning module networks that correspond to three fundamental reasoning mechanisms.

2.2 Neural Module Networks

Neural module networks is a general framework that explicitly models the compositionality of languages by decomposing the network into neural modules. It has been widely used in visual question answering and visual grounding. [Andreas et al., 2016] employs an off-the-shelf parser to parse the questions into a tree, leading to brittleness caused by parsing errors. [Hu et al., 2017] trains an RNN to decode the language into the sequence, thus requiring extra human annotations. [Hu et al., 2018] removes language parser and additional annotations by using a soft and continuous module layout. However, the case of video captioning is more complex since there are no fully observed captions during inference thus all the above methods are not applicable. To this end, we design a dynamic module selector to construct the reasoning procedure step-by-step during the generation process.

Recently, the pioneering works [Liu et al., 2018; Zha et al., 2019; Yang et al., 2019b; Tian and Oh, 2019] are trying to adopt neural module networks into image captioning. However, their modules are designed to produce several types of features and their module compositions relied on a soft attention mechanism. As a contrast, our RMN employs several sophisticated spatio-temporal reasoning modules to perform more complex visual reasoning over videos, and designs a discrete and dynamic module selector to make a selection at each step, making the generation process explicit and explainable.

3 Approach

In this section, we will describe the proposed Reasoning Module Networks (RMN) in more detail. Figure 2 gives a walk-through example of our RMN at timestep t. Our RMN can be divided into four stages: Encoding (Section 3.1), Module Reasoning (Section 3.2), Module Selection (Section 3.3), and Decoding (Section 3.4). In Section 3.5, we detail the joint training strategy for both reasoning modules and the module selector.

3.1 Encoder

For the given video of N frames, we first represent it with three types of features, i.e., appearance features Va extracted from a 2D-CNN, object features Vo extracted from a R-CNN on each frame, and motion features Vm extracted from a 3DCNN. Note that to model temporal information for those visual features, we have post-processed Va and Vm with BiLSTMs. And Vo contains an additional dimension on space.

For the previous generated words {w1, · · · , wt-1}, we encode them by a LSTM (denoted as enLSTM) which takes the global visual feature ¯ v, the last generated word embedding vector et-1, and the last timestep hidden state hde t-1 of deLSTM (cf. Eqn. (9)) as input:

It is worth noticing that since hen t encodes rich information from history, it will be used to guide the reasoning process and the module selection.

3.2 ReasoningModules

As mentioned in the Introduction, there are three fundamental reasoning mechanisms involved in video captioning. Therefore, we design three corresponding reasoning modules. In the following, we will first introduce the attention functions used in the modules, and then describe each module in detail.

Attention Functions

We follow the widely used additive attention formulation [Bahdanau et al., 2015]:

where V and q are values and queries of attention, w1, W2, and W3 are trainable parameters, 1 is an all-one vector. Note that, the attention can be executed on any dimensions, therefore, we further define AoS(·) as the Attention over the dimension on Space, and define AoT(·) as the Attention over the dimension on Time.

LOCATE Module

It is designed to generate visual words, e.g., "man" and "basketball". Generating this type of words requires the model to attend on one specific region over both space and time. Therefore, we first apply an Attention over Space (AoS) for object features Vo and then apply an Attention over Time (AoT) together with Va, formally:

where ⊕ denotes concatenate operation.

RELATE Module

It is designed to generate action words, e.g., "riding" and "shooting". Modeling action requires reasoning over both space and time. Take Figure 1 as an example, to generate the word "shooting", we must be aware that the man is holding a basketball, and then notice that the basketball is flying to the basket, finally we can inference that the man is "shooting". It means we must connect two scenes at different time. To achieve this reasoning mechanism, we insert a pairwise interaction (i.e., concatenate every possible pair of two tensors) between the AoS and AoT, formally:

where P(·, ·) denotes the pairwise interaction function that Pij (A, B) = Ai ⊕ Bj .

FUNC Module

It is designed to generate function words to complete the whole sentences. Since the function words only require language information, thus we propose to recall the history decoder cell states of deLSTM (cf. Eqn. (9)) to generate the current words by an AoT, formally:

So far, we have detailed the three proposed reasoning modules. Next, we will describe how to make a selection of those modules at each timestep.

3.3 Module Selector

As illustrated in Figure 2, the module selection consists of two steps. First, we calculate a score for each module to measure the probability of each module could be selected. Second, we sampling one determined module based on those scores with Gumbel Softmax [Jang et al., 2017] strategy.

In detail, we formulate the scoring function S(·, ·) as:

Thus we can get slt = S(hen t , vlt ), srt = S(hen t , vr t ), and sft = S(hen t , vf t ) for the three LOCATE, RELATE, and FUNC modules, respectively.

After that, the straightforward selection method is directly choosing the module with the maximum score. However, it will block the gradients off the module selector for the function arg max is non-differentiable. To this end, we apply a relaxation between forward pass and backward pass, formally:

where we take an approximation that relaxes the discrete onehot decision vector zt into a continuous ˜ zt. G is the Gumbel noise drawn from i.i.d. Gumbel(0, 1). τ is a temperature parameter to control the strength of softmax. Please refer to [Jang et al., 2017] for more mathematical details.

With this decision vector zt, we can get the final visual reasoning result as:vt = zt ⊗ [vlt , vr t , vf t ], (8)

where ⊗ denotes inner product. It is worth noticing that zt is an one-hot vector, thus only one determined module will be selected in the forward pass.

3.4 Decoder

For each timestep t, we decode the final visual reasoning result by a LSTM (denotes as deLSTM):

Recall that the hidden and cell states were used in Eqn. (1) and Eqn. (5). Further, we decode the word probability pt as:

where MLP is two-layers with tanh as activation function.

3.5 End-to-End Training

Thanks to the Gumbel-Softmax strategy, our RMN model can be trained in an end-to-end manner. Given a video and the corresponding ground-truth caption {w∗ t } for t ∈ [1, T], where T is the sentence length, the objective is to minimize the cross-entropy loss function given by:

Further, to ensure the module selector sticking on the linguistic structure, we propose to supervise the module selection by extra Part-of-Speech (POS) tag labels. Accordingly, we apply a Kullback-Leibler Divergence (KLD) loss to make the predicted module decision vector ˜ z and the ground-truth tag distribution S∗ = {s∗t }, t ∈ [1, T] as close as possible, formally:

To collect the ground-truth S∗ , we first detected POS tag labels by Spacy Tagging Tool 1 , and then assigned [NN*] and [JJ*] to the LOCATE module, [VB*] to the RELATE module, and the other to the FUNC module.

Therefore, the overall loss function of our RMN model is given by:

where λ is a trade-off weight.

4 Experiments

4.1 Datasets andMetrics

We conduct experiments on two widely used video captioning datasets with several standard evaluation metrics to verify the effectiveness of our proposed method.

Datasets

MSVD. The MSVD dataset [Chen and Dolan, 2011] consists of 1,970 short video clips selected from Youtube, where each one depicts a single activity in the open domain, and each video clip is annotated with multi-lingual captions. Since we only consider the English captions in this paper, each video clip has roughly 41 descriptions. To be consistent with previous works, we split the dataset to 3 subsets, 1,200 clips for training, 100 clips for validation, and the remaining 670 clips for testing.

MSR-VTT. The MSR-VTT [Xu et al., 2016] is a large-scale dataset for the open domain video captioning, which consists of 10,000 video clips from 20 categories, and each video clip is annotated with 20 English sentences by Amazon Mechanical Turks. There are about 29,000 unique words in all captions. Following the existing works, we use the standard splits, namely 6,513 clips for training, 497 clips for validation, and 2,990 clips for testing.

Evaluation Metrics

We use several widely used automatic evaluation metrics to evaluate the quality of the generated captions, i.e., BLEU [Papineni et al., 2002], METEOR [Banerjee and Lavie, 2005], CIDEr [Vedantam et al., 2015], ROUGE-L [Lin, 2004]. Most of these metrics are originally proposed for machine translation or image captioning, the higher score indicates better quality of the captions.

4.2 Implementation Details

Dataset Preprocessing

We first convert all captions to lower case and remove punctuations, then we truncate the captions with more than 26 words and zero pad the captions with less than 26 words. The vocabulary size is set to 7,351 for MSVD and 9,732 for MSR-VTT with removing the words appear less than twice and five times respectively.

1 https://spacy.io

Feature Extraction

In our experiments, we use InceptionResNetV2 (IRV2) [Szegedy et al., 2017] as 2D CNN and I3D [Carreira and Zisserman, 2017] as 3D CNN to extract appearance features and motion features respectively, then we equally-spaced 26 features for each video. The IRV2 is trained on ILSVRC-2012CLS image classification dataset [Russakovsky et al., 2015] and the I3D is trained on Kinetics action classification dataset [Kay et al., 2017]. We adopt Faster-RCNN [Ren et al., 2016] which is trained by [Anderson et al., 2018] to extract 36 region features for each frame (26 equally-spaced frames for each video).

Training Details

Our model is optimized by Adam Optimizer [Kingma and Ba, 2015], the initial learning rate is set to 1e-4. For the MSVD dataset, the hidden size of the LSTM is 512 and the learning rate is divided by 10 every 10 epochs. For the MSRVTT dataset, the hidden size of the LSTM is 1,300 and the learning rate is divided by 3 every 5 epochs. During testing, we use beam search with size 2 for the final caption generation.

4.3 Ablation Study

As shown in Table 1, we compare our RMN against a set of other ablated models with various settings: (1) RMN (LOCATE): the model that only deployed with Locate module; (2) RMN (RELATE): the model that only deployed with Related module; (3) RMN (S): the model that softly fuses three modules, i.e., zt in Eqn. (8) is computed as softmax(log([slt , srt , sft ]); (4) RMN (H): the model that hard selects one of three modules; (5) RMN (S+L): the model that softly fuses three modules and trained by linguistic loss; (6) RMN (H+L): the model that hard selects one of three modules and trained by linguistic loss. According to the results shown in Table 1, we have the following observations.

Effect of Reasoning Modules

RMN (RELATE) consistently outperforms RMN (LOCATE) on both MSVD and MSR-VTT. This is because the RELATE module models much richer visual action information than LOCATE, leading to performance improvement over RMN (LOCATE).

RMN (S) outperforms RMN (LOCATE) and RMN (RELATE) under most metrics. This indicates that by fusing different types of reasoning mechanisms, the model can generate better video descriptions.

Effect of Gumbel Strategy

Comparing RMN (H) with RMN (LOCATE) and RMN (RELATE), we observe opposite results on MSVD and MSRVTT. To figure out the reasons for this observation, we count the proportion of the three modules occupied in Figure 3. We can find that without the linguistic loss, RMN (H) only employs two modules, i.e., LOCATE module and RELATE module, indicating the failure of training.

Therefore, the reason for the opposite results is that RMN (H) failed in training on MSVD while succeeded on MSRVTT. Actually, even though Gumbel strategy can relax the discrete decision to continuous decision, it still requires large training examples. Therefore, RMN (H) underperforms RMN (LOCATE) and RMN (RELATE) on MSVD which only contains 1,200 training samples, while outperforming them on MSR-VTT which contains 6,513 training samples.

Similarly, even though RMN(S) surpasses RMN(H) under some metrics, but if we employ the linguistic loss to make GT: there is a man riding a vehicle into the forest

RMN: a man is riding a vehicle in a forest

(a) MSR-VTT: 7393

GT: a man is playing a flute

RMN: a man is playing a flute

(d) MSVD: z0zb--BOhDY_16_22

GT: a group of boys are playing basketball with a coach

RMN: a group of people are playing basketball

(b) MSR-VTT: 7458

GT: a man and a woman are speaking on the phone

RMN: two people are talking on the phone

(e) MSVD: q3I3R_gqy8M_34_37

GT: there is a man in blue is swimming in the sea

RMN: a man is swimming in the sea and talking about the sea

(c) MSR-VTT: 7505

GT: someone is pouring tomato sauce into a pot of meat

RMN: a man is pouring sauce into a pot

(f) MSVD: hJFBXHtxKIc_204_209

sure both soft fusion and hard module selector works well, RMN (H+L) outperforms RMN (S+L) with a margin.

Effect of Linguistic Loss

Comparing RMN (H+L) with RMN (H), as well as comparing RMN (S+L) with RMN (S), we find that the linguistic loss consistently improves the performance, indicating the importance of linguistic information for video captioning.

We can also observe the consistent results in Figure 3. By imposing the linguistic loss, we can enforce the module selector sticking with the linguistic structure, and the proportion of each module is approaching to the ground-truth.

Further, we carried out the word cloud statistics as shown in Figure 4. We can find that the three modules have a strong pattern: the LOCATE module mainly generates visual words, e.g., "man" and "person"; the RELATE module mainly generates action words, e.g., "playing" and "talking"; the FUNC module mainly generates function words, e.g., "about" and "to". It indicates that the model has learned linguistic knowledge correctly.

4.4 Comparison with State-of-the-Art

We compare our proposed RMN with the most recent stateof-the-art methods on MSVD and MSR-VTT datasets. According to whether they leverage the POS labels, we categorize them into two groups: 1) traditional encoder-decoder based models MAM-RNN [Li et al., 2017], RecNet [Wang et al., 2018], MARN [Pei et al., 2019], and OA-BTG [Zhang and Peng, 2019], and 2) POS strengthened model POSCG [Wang et al., 2019] and Mixture [Hou et al., 2019].

As shown in Table 2, we can find that: 1) the methods that leverage POS labels outperform the methods without POS information, and 2) our proposed reasoning module network outperforms the methods with POS labels on most metrics and achieves new state-of-the-art. It is worth noting that CIDEr is proposed for captioning task specifically, and is considered more consistent with human judgment. Our model achieves the best CIDEr score on both datasets, which demonstrates our RMN model can generate captions that more in line with human descriptions.

4.5 Qualitative Analysis

In this section, we would like to investigate the generation process of our model by qualitative results. Here we provide some video captioning examples in Figure 5. As expected, our module selection is explicit and reasonable, for example, we compose "man", "flute", and "phone" with the LOCATE module, "playing", "riding", and "talking" with RELATE module, and "a", "of", and "the" with FUNC module.

The above observations suggest that the generation process of the proposed RMN is totally explicit, since at each timestep the module selects one determined module to produce the current word. In addition, each module is welldesigned to perform spatio-temporal visual reasoning, e.g., the spatio-temporal attention for LOCATE module and pairwise interaction reasoning over both space and time for RELATE module, indicating that our model is explainable.

5 Conclusion

In this paper, we proposed a novel reasoning neural module networks (RMN) for video captioning that performs visual reasoning on each step along the generation process. Specifically, we designed three sophisticated reasoning modules for spatio-temporal visual reasoning. To dynamically compose the reasoning modules, we proposed a discrete module selector which is trained by a linguistic loss with a Gumbel approximation. Extensive experiments verified the effectiveness of the proposed RMN, and the qualitative results indicated the caption generation process is explicit and explainable.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant 2017YFB1300201, the National Natural Science Foundation of China (NSFC) under Grants U19B2038, 61620106009 and 61725203 as well as the Fundamental Research Funds for the Central Universities under Grant WK2100100030.