Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning

1 Introduction

Previously, visual content understanding [Song et al., 2016; Gao et al., 2017] and natural language processing (NLP) are not correlative with each other. Integrating visual content with natural language learning to generate descriptions for images, especially for videos, has been regarded as a challenging task. Video captioning is a critical step towards machine intelligence and many applications in daily scenarios, such as video retrieval [Wang et al., 2017; Song et al., 2017], video understanding, blind navigation and automatic video subtitling.

∗Corresponding author: Heng Tao Shen, Lianli Gao

Thanks to the rapid development of deep Convolutional Neural Network (CNN), recent works have made significant progress for image captioning [Vinyals et al., 2015; Xu et al., 2015; Lu et al., 2016; Karpathy et al., 2014; Fang et al., 2015; Chen and Zitnick, 2014; Chen et al., 2016]. However, compared with image captioning, video captioning is more difficult due to the diverse sets of objects, scenes, actions, attributes and salient contents. Despite the difficulty there have been a few attempts for video description generation [Venugopalan et al., 2014; 2015; Yao et al., 2015; Li et al., 2015; Gan et al., 2016a], which are mainly inspired by recent advances in translating with Long Short-Term Memory (LSTM). The LSTM is proposed to overcome the vanishing gradients problem by enabling the network to learn when to forget previous hidden states and when to update hidden states by integrating memory units. LSTM has been successfully adopted to several tasks, e.g., speech recognition, language translation and image captioning [Cho et al., 2015; Venugopalan et al., 2014]. Thus, we follow this elegant recipe and choose to extend LSTM to generate the video sentence with semantic content.

Early attempts were proposed [Venugopalan et al., 2014; 2015; Yao et al., 2015; Li et al., 2015] to directly connect a visual convolution model to a deep LSTM networks. For example, Venugopalan et al. [Venugopalan et al., 2014] translate videos to sentences by directly concatenating a deep neural network with a recurrent neural network. More recently, attention mechanism [Gu et al., 2016] is a standard part of the deep learning toolkit, contributing to impressive results in neural machine translation [Luong et al., 2015], visual captioning [Xu et al., 2015; Yao et al., 2015] and question answering [Yang et al., 2016]. Visual attention models for video captioning make use of video frames at every time step, without explicitly considering the semantic attributes of the predicted words. For example, in Fig. 1, some words (i.e., "man", "shooting" and "gun") belong to visual words which have corresponding canonical visual signals, while other words (i.e., "the", "a" and "is") are non-visual words, which require no visual information but language context information [Lu et al., 2016]. In other words, current visual attention models make use of visual information for generating each work, which is unnecessary or even misleading. Ideally, video description not only requires modeling and integrating their sequence dynamic temporal attention information into a natural language but also needs to take into account the relationship between sentence semantics and visual content [Gan et al., 2016b], which to our knowledge has not been simultaneously considered.

To tackle these issues, inspired by the attention mechanism for image captioning [Lu et al., 2016], in this paper we propose a unified encoder-decoder framework (see Fig. 1), named hLSTMat, a Hierarchical LSTMs with adjusted temporal attention model for video captioning. Specifically, first, in order to extract more meaningful spatial features, we adopt a deep neural network to extract a 2D CNN feature vector for each frame. Next, we integrate a hierarchical LSTMs consisting of two layers of LSTMs, temporal attention and adjusted temporal attention to decode visual information and language context information to support the generation of sentences for videos description. Moreover, the proposed novel adjusted temporal attention mechanism automatically decides whether to rely on visual information or not. When relying on visual information, the model enforces the gradients from visual information to support video captioning, and decides where to attend. Otherwise, the model predicts the words using natural language model without considering visual signals.

It is worthwhile to highlight the main contributions of this proposed approach: 1) We introduce a novel hLSTMat framework which automatically decides when and where to use video visual information, and when and how to adopt the language model to generate the next word for video captioning. 2) We propose a novel adjusted temporal attention mechanism which is based on temporal attention. Specifically, the temporal attention is used to decide where to look at visual information, while the adjusted temporal model is designed to decide when to make use of visual information and when to rely on language model. A hierarchical LSTMs is designed to obtain low-level visual information and high-level language context information. 3) Experiments on two benchmark datasets demonstrate that our method outperforms the state-of-the-art methods in both BLEU and METEOR.

2 The Proposed Approach

In this section, first we briefly describe how to directly use the basic Long Short-Term Memory (LSTM) as the decoder for video captioning task. Then we introduce our novel encoderdecoder framework, named hLSTMat (see Fig. 1).

2.1 A Basic LSTM for Video Captioning

To date, modeling sequence data with Recurrent Neural Networks (RNNs) has shown great success in the process of machine translation, speech recognition, image and video captioning [Chen and Zitnick, 2014; Fang et al., 2015; Venugopalan et al., 2014; 2015] etc. Long Short-TermMemory (LSTM) is a variant of RNN to avoid the vanishing gradient problem [Bengio et al., 1994].

LSTM Unit. A basic LSTM unit consists of three gates (input it, forget ft and output ot), a single memory cell mt. Specifically, it allows incoming signals to alter the state of the memory cell or block it. ft controls what to be remembered or be forgotten by the cell, and somehow can avoid the gradient from vanishing or exploding when back propagating through time. Finally, ot allows the state of the memory cell to have an effect on other neurons or prevent it. Basically, the memory cell and gates in a LSTM block are defined as follows:

where the weight matrices W, U, and b are parameters to be learned. yt represents the input vector for the LSTM unit at each time t. σ represents the logistic sigmoid nonlinear activation function mapping real numbers to (0, 1), and it can be thought as knobs that LSTM learns to selectively forget its memory or accept the current input. φ denotes the hyperbolic tangent function tanh. ⊙ is the elementwise product with the gate value. For convenience, we denote ht,mt = LSTM(yt, ht-1,mt-1) as the computation function for updating the LSTM internal state.

Video Captioning. Given a video input x, an encoder network φE encodes it into a continuous representation space:

where φE usually denotes a CNN neural network, n denotes the number of frames in x, vi ∈ Rd is the frame-level feature of the i-th frame, and it is d-dimensional. Here, LSTM is chosen as a decoder network φD to model V to generate a description z = {z1, · · · , zT} for x, where T is the description length. In addition, the LSTM unit updates its internal state ht and the t-th word zt based on its previous internal state ht-1, the previous output yt and the representation V:

In addition, the LSTM updates its internal state recursively until the end-of-sentence tag is generated. For simplicity, we named this simple method as basic-LSTM.

2.2 Hierarchical LSTMs with Adjusted Temporal Attention for Video Captioning

In this subsection, we introduce our hLSTMat framework, which consists of two components: 1) a CNN Encoder and 2) an attention based hierarchical LSTM decoder.

CNN Encoders

The goal of an encoder is to compute feature vectors that are compact and representative and can capture the most related visual information for the decoder. Thanks to the rapid development of deep convolutional neural networks (CNNs), which have made a great success in large scale image recognition task [He et al., 2016], object detection [Ren et al., 2015] and visual captioning [Venugopalan et al., 2014]. High-level features can been extracted from upper or intermediate layers of a deep CNN network. Therefore, a set of well-tested CNN networks, such as the ResNet-152 model [He et al., 2016] which has achieved the best performance in ImageNet Large Scale Visual Recognition Challenge, can be used as a candidate encoder for our framework.

Attention based Hierarchical Decoder

Our decoder (see Fig. 2) integrates two LSTMs. The bottom LSTM layer is used to efficiently decode visual features, and the top LSTM is focusing on mining deep language context information for video captioning. We also incorporate two attention mechanisms into our framework. A temporal attention is to guide which frame to look, while the adjusted temporal attention is proposed to decide when to use visual information and when to use sentence context information. The top MLP layer is to predict the probability distribution of each word in the vocabulary.

Unlike vanilla LSTM decoder, which performs mean pooling over 2D features across each video to form a fixeddimension representation, attention based LSTM decoder is focusing on a subset of consecutive frames to form a fixeddimensional representation at each time t.

• Bottom LSTM Layer. For the bottom LSTM layer, the updated internal hidden state depends on the current word yt, previous hidden state ht-1 and memory state mt-1:

where yt = E[yt] denotes a word feature of a single word yt. Mean(·) denotes a mean pooling of the given feature set vi. Wih and Wic are parameters that need to be learned.

• Top LSTM Layer. For the top LSTM, it takes the output of the bottom LSTM unit output ht, previous hidden state ¯ht-1 and the memory state ¯mt-1 as inputs to obtain the hidden state ¯ht at time t, and it can be defined as below:

• Attention Layers. In addition, for attention based LSTM, context vector is in general an important factor, since it provides meaningful visual evidence for caption generation [Yao et al., 2015]. In order to efficiently adjust the choose of visual information or sentence context information for caption generation, we defined an adjusted temporal context vector ¯ct and a temporal context vector ct at time t. See below:

where ψ denotes the function of our adjust gate, while ϕ denotes the function of our temporal attention model. Moreover, ¯ct denotes the final context vector through our adjusted gate, and ct represents intermediate vectors calculated by our temporal attention model. These two attention layers will be described in details in Sec. 2.3 and Sec. 2.4.

• MLP layer. To output a symbol zt, a probability distribution over a set of possible words is obtained using ht and our adjusted temporal attention vector ¯ct:

where Up, Wp, bp and d are parameters to be learned. Next, we can interpret the output of the softmax layer pt as a probability distribution over words:

where V denotes the features of the corresponding input video, and Θ are model parameters.

To learn Θ in our modal, we minimize the negative logarithm of the likelihood:

where T denotes the total number of words in sentence. Therefore, Eq.8 is regarded as our loss function to optimize our model.

After the parameters are learned, we choose BeamSearch [Vinyals et al., 2015] method to generate sentences for videos, which iteratively considers the set of the k best sentences up to time t as candidates to generate sentence of time t + 1, and keeps only best k results of them. Finally, we approximates D = argmaxD′ P r(D′ |X) as our best generated description. In our entire experiment, we set the beam size of BeamSearch as 5.

2.3 Temporal AttentionModel

As mentioned above, context vector ct is an important factor in encoder-decoder framework. To deal with the variability of the length of videos, a simple strategy [Venugopalan et al., 2014] is used to compute the average of features across a video, and this generated feature is used as input to the model at each time step:

However, this strategy effectively collapses all frame-level features into a single vector, neglecting the inherent temporal structure and leading to the loss of information. Instead of using a simple average strategy (see Eq. 9), we wish to take the dynamic weight sum of the temporal feature vectors according to attention weights αit , which are calculated by a soft attention. For each vi at t time step, we use the follow function to calculate ct:

where at t time step ∑ni=1 αit = 1.

In this paper, we integrate two LSTM layers, a novel temporal attention model for computing the context vector ct in Eq. 5 proposed in our framework. Given a set of video features V and the current hidden state of the bottom layer LSTM ht, we feed them into a single neural network layer, and it returns an unnormalized relevant scores εt. Finally, a softmax function is applied to generate the attention distribute over the n frames of the video:

where wT , Wa, Ua and ba are parameters to be learned. αt ∈ Rn is the attention weight which quantifies the relevance of features in V.

Different from [Yao et al., 2015], we utilize the current hidden state instead of previous hidden state ht generated by the first LSTM layer to obtain the context vector ct, which focuses on salient feature in the video.

2.4 Adjusted Temporal AttentionModel

In this paper we propose an adjusted temporal attention model to compute a context vector ¯ct in Eq. 5, shown in Fig. 2, to make sure that a decoder uses nearly no visual information from video frames to predict the non-visual words, and use the most related visual information to predict visual words. In our hierarchical LSTM network, the hidden state in the bottom LSTM layer is a latent representation of what the decoder already knows. With the hidden state ht, we extend our temporal attention model, and propose an adjusted model that is able to determine whether it needs to attend the video to predict the next word. In addition, a sigmoid function is applied to the hidden state ht to further filter visual information.

whereWs denotes the parameters to be learned and βt is adjusted gate at each time t. In our adjusted temporal attention model, βt is projected into the range of [0, 1]. When βt = 1, it indicates that full visual information is considered, while when βt = 0 it indicates that none visual information is considered to generate the next word.

3 Experiments

We evaluate our algorithm on the task of video captioning. Specifically, we firstly study the influence of CNN encoders. Secondly, we explore the effectiveness of the proposed components. Next, we compare our results with the state-of-theart methods.

3.1 Datasets

We consider two publicly available datasets that have been widely used in previous work.

The Microsoft Video Description Corpus (MSVD). This video corpus consists of 1,970 short video clips, approximately 80,000 description pairs and about 16,000 vocabulary words [Chen and Dolan, 2011]. Following [Yao et al., 2015; Venugopalan et al., 2015], we split the dataset into training, validation and testing set with 1,200, 100 and 670 videos, respectively.

MSR Video to Text (MSR-VTT). In 2016, Xu et al. [Xu et al., 2016] proposed a currently largest video benchmark for video understanding, and especially for video captioning. Specifically, this dataset contains 10,000 web video clips, and each clip is annotated with approximately 20 natural language sentences. In addition, it covers the most comprehensive categorizes (i.e., 20 categories) and a wide variety of visual content, and contains 200,000 clip-sentence pairs.

3.2 Implementation Details

Preprocessing

For MSVD dataset, we convert all descriptions to lower cases, and then use wordpunct tokenizer method from NLTK toolbox to tokenize sentences and remove punctuations. Therefore, it yields a vocabulary of 13,010 in size for the training split. For MSR-VTT dataset, captions have been tokenized, thus we directly split descriptions using blank space, thus it yields a vocabulary of 23,662 in size for training split. Inspired by [Yao et al., 2015], we preprocess each video clip by selecting equally-spaced 28 frames out of the first 360 frames and then feeding them into a CNN network proposed in [He et al., 2016]. Thus, for each selected frame we obtain a 2,048-D feature vector, which are extracted from the pool5 layer.

Training details

In the training phase, in order to deal with sentences with arbitrary length, we add a begin-of-sentence tag <BOS> to start each sentence and an end-of-sentence tag <EOS> to end each sentence. In the testing phase, we input <BOS> tag into our attention-based hierarchical LSTM to trigger video description generation process. For each word generation, we choose the word with the maximum probability and stop until we reach <EOS>.

In addition, all the LSTM unit sizes are set as 512 and the word embedding size is set as 512, empirically. Our objective function Eq. 8 is optimized over the whole training videosentence pairs with mini-batch 64 in size of MSVD and 256 in size of MSR-VTT. We adopt adadelta [Zeiler, 2012], which is an adaptive learning rate approach, to optimize our loss function. In addition, we utilize dropout regularization with the rate of 0.5 in all layers and clip gradients element wise at 10. We stop training our model until 500 epochs are reached, or until the evaluation metric does not improve on the validation set at the patience of 20.

Evaluation metrics

To evaluate the performance, we employ two different standard evaluation metrics: BLUE [Papineni et al., 2002] and METEOR [Banerjee and Lavie, 2005].

3.3 The Effect of Different CNN Encoders

To date, there are 6 widely used CNN encoders including C3D, GoogleNet, Inception-V3, ResNet-50, ResNet-101 and ResNet-152 to extract visual features. In this sub-experiment, we study the influence of different versions of CNN encoders on our framework. The experiments are conducted on the MSVD dataset, and the results are shown in Tab. 1. By observing Tab. 1, we find that by taking ResNet-152 as the visual decoder, our method performs best with 82.9% B@1, 72.2% B@2, 63.0% B@3, 53.0% B@4 and 33.6% METEOR, while Inception-v3 is a strong competitor, with 82.7% B@1, 72.0% B@2, 62.5% B@3, 51.9% B@4 and 33.5%METEOR. However, the gap between ResNet-152 and Inception-v3 is very small.

3.4 Architecture Exploration and Comparison

In this sub-experiment, we explore the impact of three proposed components, including basic LSTM proposed in Sec.2.1 (basic LSTM), hLSTMt which removes the adjusted mechanism from the hLSTMat, and hLSTMat, as well as comparing them with the state of the art methods: MP-LSTM [Venugopalan et al., 2014] and SA [Yao et al., 2015]. In order to conduct a fair comparison, all the methods take ResNet152 as the encoder. We conduct the same experiments on the MSVD dataset. The experimental results are shown in Tab. 2. It shows that our hLSTMat achieves the best results in all metrics with 82.9% B@1, 72.2% B@2, 63.0% B@3, 53.0% B@4 and 33.6% METEOR. Also, by comparing with SA which take previous hidden state to calculate temporal attention weight, our hLSTMt performs better for video captioning. Moreover, by comparing with hLSTMt, we find that adjusted attention mechanism can improve the performance of video captioning. We also add one-layer LSTM and adjusted attention as an additional baseline. Results show that the adjusted attention mechanism can improve the performance.

3.5 Compare with the-state-of-the-art Methods

Results on MSVD dataset

In this subsection, we show the comparison of our approach with the baselines on the MSVD dataset. Some of the above baselines only utilize video features generated by a single deep network, while others (i.e., S2VT, LSTM-E and p-RNN) make uses of both single network and multiple network generated features. Therefore, we first compare our method with approaches using static frame-level features extracted by a single network. In addition, we compare our method with methods utilized different deep features or their combinations. The results are shown in Tab.3. When using static frame-level features, we have the following observations:

1. Compared with the best counterpart (i.e., p-RNN) which only takes spatial information, our method has 8.7% improvement on B@4 and 2.5% on METEOR.

2) The hierarchical structure in HRNE reduces the length of input flow and composites multiple consecutive input at a higher level, which increases the learning capability and enables the model encode richer temporal information of multiple granularities. Our approach (53.0% B@4, 33.6% METEOR) performs better than HRNE (43.6% B@4, 32.1% METEOR) and HRNE-SA (43.8% B@4, 33.1% METEOR). This shows the effectiveness of our model.

3) Our hLSTMat (53.0% B@4, 33.6% METEOR) can achieve better results than our hLSTMt (52.1% B@4, 33.3% METEOR). This indicates that it is beneficial to incorporate the adjusted temporal attention into our framework.

On the other hand, utilizing both spatial and temporal video information can enhance the video caption performance. VGGNet and GoogleNet are used to generate spatial information, while optical flow and C3D are used for capturing temporal information. For example, compared with LSTM-E(V) and LSTM-E (C), LSTM-E(V+C) achieves higher 45.3% B@4 and 31.0%METEOR. In addition, for p-RNN, p-RNN(V+C) (49.9% B@4 and 32.6% METEOR ) performs better than both p-RNN(V) (44.3% B@4 and 31.3% METEOR) and pRNN(C) (47.4% B@4 and 30.3% METEOR).

Our approach achieves the best results (53.0% B@4 and 33.6%METEOR) using static frame-level features compared with approaches combining multiple deep features. For S2VT(V+O), LSTM-E(V+C) and p-RNN(V+C), they use two networks VGGNet/GoogleNet and optical flow/C3D to capture video's spatial and temporal information, respectively. Compared with them, our approach only utilizes ResNet-152 to capture frame-level features, which proves the effectiveness of our hierarchical LSTM with adjusted temporal attention model.

We adopt questionnaires collected from ten users with different academic backgrounds. Given a video caption, users are asked to score the following aspects: 1) Caption Accuracy, 2) Caption Information Coverage, 3) Overall Quality. Results show that our method outperforms others at 'Overall Quality', and 'Caption Accuracy' with small margin. But it has lower value for 'Information coverage' than p-RNN.

Results on MSR-VTT dataset

We compare our model with the state-of-the-art methods on the MSR-VTT dataset, and the results are shown in Tab. 4. Our model performs the best on all metrics, with 38.3% @B4 and 26.3% METEOR. Compared with our methods using only temporal attention, the performance is improved by 1.1% for @B4, and 0.2% for METEOR. This verifies the effectiveness of our method.

4 Conclusion and FutureWork

In this paper, we introduce a novel hLSTMat encoder-decoder framework, which integrates a hierarchical LSTMs, temporal attention and adjusted temporal attention to automatically decide when to make good use of visual information or when to utilize sentence context information, as well as to simultaneously considering both low-level video visual features and language context information. Experiments show that hLSTMat achieves state-of-the-art performances on both MSVD and MSR-VTT datasets. In the future, we consider incorporating our method with both temporal and visual features to test the performance.

Acknowledgments

This work is supported by the Fundamental Research Funds for the Central Universities (Grant No. ZYGX2014J063, No. ZYGX2014Z007) and the National Natural Science Foundation of China (Grant No. 61502080, No. 61632007, No. 61602049).