Video Captioning with Tube Features

1 Introduction

Vision and language are two most important ways for human beings to perceive the world. Video captioning is one of the techniques that bridge them together [Pan et al., 2016a; Li et al., 2017]. Specifically, the goal of video captioning is to describe the video content with natural language text. It is an important task with wide applications, such as text-based video retrieval, human-robot interaction and so on.

Earlier works on video captioning are mostly templatebased, which first detect the semantic content (i.e., subjective, verb, objective), and then generate the caption based on a sentence template [Guadarrama et al., 2013; Thomason et al., 2014]. However, these template-based approaches are inferior in modeling the rich information of natural language. Recently, inspired by the great success of Recurrent Neural Network (RNN) in language modeling [Luong et al., 2015; Kalchbrenner and Blunsom, 2013], RNN-based approaches are introduced to video captioning and have achieved exciting results [Venugopalan et al., 2015b]. Generally, given the visual feature as input, RNN is utilized to generate the caption word by word.

Figure 1: Tube features capture the dynamic information of objects. Moreover, the proposed approach can adaptively attend to the most correlated tubes when generating a certain word. The tubes bounded in different colors denote the most emphasized tube when generating the word in the same color.

As the caption is automatically generated based on the video content, visual feature is quite important to the caption quality [Baraldi et al., 2017; Pan et al., 2016a]. Initially, the visual feature for video captioning is directly extended from the image captioning task, i.e., average pooling the deep frame features [Venugopalan et al., 2015b]. Later, to extract more correlated visual features, attention models are developed to selectively focus on a subset of video frames [Long et al., 2016; Hori et al., 2017; Li et al., 2017]. However, these feature extraction methods focus on the global information, but the importance of salient objects is overlooked, which may restrict the accuracy of generated captions. Based on this point, in this paper, we design tube features to extract object information for the task of video captioning.

Actually, as depicted in the bottom of Figure 1, a tube reflects the trajectory of corresponding object. Firstly, FasterRCNN [Ren et al., 2015] is employed to detect the objects in each frame. Then, these object regions are sequentially linked to be tubes. Besides, to avoid the loss of global frame information, all frames are linked as an additional tube. After tubes are generated, an encode-decoder architecture is designed to construct our network, where the encoder is a bidirectional LSTM, which is utilized to capture the dynamic information of objects by encoding each tube feature into a fix-sized visual vector, and the decoder LSTM is a single LSTM, used to generate the caption word by word. Specifically, at each time step, all the visual vectors and the text feature of previous word are input to the decoder LSTM. Besides, an attention model is designed to adaptively emphasize different tubes when generating a certain word.

To our knowledge, this is the first approach that extracts object-aware features for the task of video captioning. It is capable of reducing the interference caused by irrelevant background information and improving the accuracy of generated captions. Moreover, the experimental results on the MSVD dataset [Guadarrama et al., 2013] and Charades dataset [Sigurdsson et al., 2016] have verified the effectiveness of the proposed approach.

2 Our Approach

In this paper, we develop a video caption generator conditioning on tube features. In the following, we first present the procedure of tube construction, and then introduce the generation of video caption.

2.1 Tube Generation

In general, tubes are formed by the trajectories of objects. Firstly, the Faster-RCNN [Ren et al., 2015] is employed to detect objects in the video. After that, tubes are generated by sequentially linking the detected object regions (i.e., bounding box). In other words, each object tube is composed of those regions in different frames but belonging to the same object.

After bounding boxes are detected, a similarity graph is constructed among them. As depicted in Figure 2, the edge between each pair of bounding boxes in adjacent frames are labeled with a similarity score which indicates the confidence that they belong to the same object. Without loss of generality, suppose that rj and rj+1 are two regions in adjacent frames j and j + 1, the similarity score s is computed as:

where s is a comprehensive index that jointly considers the relationships of distance, area, and Intersection Overlap Union (IOU) between region rj and rj+1, and λ is a scalar to normalize s to (0, 1). Specifically, the three terms on the right of Equ. (1) are defined as follows:

1) Dis ind. It is designed to exploit the location relationship between two regions, a higher Dis ind indicates that two regions are closer to each other. The distance of two regions is determined by the Euclidean distance of their geometrical center, denoted as dis (rj, rj+1). Then, Dis ind is defined as:

where σ denotes the maximum Euclidean distance among regions of frame j and j + 1.

2) Area ind. It represents the difference between the area sizes of two regions, a higher Area ind means two regions have similar size. It is formulated as:

where area (·) denotes the region size.

3) Iou ind. It stands for the rate of the overlapping area between two regions, which is defined as:

After the similarity graph is constructed, the tubes are generated by finding the optimal path in the graph, which is formulated as:

where tubei = [r1, r2, . . . , rJ] is the linked region sequence for tube i. In Equ. (5), the tubes are generated iteratively by dynamic programming [Gkioxari and Malik, 2015]. When one tube is generated, the edges correlated to these regions are removed from the similarity graph. Specifically, Si denotes the similarity graph at the t-th iteration. Equ. (5) is repeated until the region set is empty, and the object tubes are obtained finally. Besides, to include the global information of frames, the frame features are taken as an additional tube and added to the tube set.

After tubes generated, a bidirectional LSTM is employed to capture the dynamic information by encoding each tube into a fix-sized visual vector. In fact, the bidirectional LSTM consists of a forward LSTM and a backward LSTM, where the main difference is that the backward LSTM operates reversely. In this case, the encoding can be formulated as:

where LSTM (·) is the short form of the calculations in each LSTM unit [Hochreiter and Schmidhuber, 1997]. hfj and hbj denote the hidden state of the forward LSTM and backward LSTM, respectively. Finally, the tube feature τ i is defined as the concatenation of the final forward and backward hidden state, i.e.,

where [·, ·] stands for the vector concatenation operation.

2.2 Video Caption Generation

After the encoding of tube features, another LSTM is employed as the decoder to generate the video caption. At each time step, the hidden layer of the decoder LSTM is generated by:

where vt denotes the input visual feature at time step t, and et-1 is the text feature of previous word. Given the hidden state ht , a probability distribution over the vocabulary is calculated by:

where Wp, Up and bp are the training parameters. pt denotes the predicted probability of each word in the vocabulary to be selected, its dimensionality is equal to the vocabulary size. Once Equ. (10) is computed, the t-th word of the caption is selected from the vocabulary by the maximum value in pt.

It can be observed from Equ. (10) that pt is determined jointly by current visual feature vt, previous text feature et-1, together with the history information captured in ht. In this paper, the text feature is extracted for each word by embedding its one-hot feature into the same space as the visual feature, and the visual feature is generated by applying attention model to the tube visual vectors {τ i}ni=1 , here n denotes the number of tubes.

Essentially, the attention model is designed to make the decoder LSTM adaptively attend to the most correlated visual features. At each time step, the visual feature input to the LSTM (i.e., vt in Equ. (9)) is a weighted combination of the tube visual vectors,

where αit stands for the attention weight of the i-th tube when generating the t-th word of the caption. It can be observed from Equ. (11) that a higher attention weight means the corresponding tube is more emphasized in composing the visual feature.

According to [Yao et al., 2015], αit represents the relevance of the i-th tube given all the words generated previously, i.e., e1, . . . , et-1. Fortunately, except et-1, they are recorded by the previous hidden state ht-1 of the decoder LSTM. Thus, the relevance score is calculated by:

where w, Wα and bα are the parameters to be learned, and

Then, the attention weight αit is obtained after normalizing li t by:

Note that αit is a dynamic attention weight that changes in different time steps, so that our attention model can adaptively focus on different tubes when generating each word.

3 Experiments

3.1 Setup

Datasets

The MSVD [Guadarrama et al., 2013] dataset contains 1970 video clips collected from YouTube. These clips are opendomain, including different scenes, activities and scenarios, etc. In total, there are about 80839 video captions annotated by Mechanical Turkers. On average, each video contains 41 captions, and each caption has about 8 words. In this paper, the dataset is split into three sets, 1200 videos for training, 100 videos for validation, and the remaining 670 videos for testing.

The Charades dataset [Sigurdsson et al., 2016] is much more challenging, which is composed of 9848 videos, and each video records a sequence of daily living activities in indoor environment, including cooking, cleaning, watching TV, etc. Exactly, this dataset provides 16130 video captions, and each caption contains 23 words on average. In this paper, the dataset is split into a training set of 7569 videos, a validation set of 400 videos, and a testing set of 1863 videos.

Feature Extraction

In this paper, the Faster-RCNN is utilized to detect objects and extract their features. Practically, to analyze the influence of different features, the Faster-RCNN is trained on PASCAL VOC 2007 and MicroSoft COCO 2014 on the basis of VGG16 [Simonyan and Zisserman, 2014] and Resnet-101 [He et al., 2016], respectively.

To extract text features, we first build the vocabulary for each dataset. In this paper, all the captions are processed by tokenizing the sentences, lowercasing the words, and removing rare words. After that, we obtain a vocabulary of 2743 words for the MSVD dataset, and 1525 words for the Charades dataset, respectively. Finally, the words are extracted with one-hot features (1-of-N coding, N denotes the vocabulary size) and then embedded into 300-dimensional GloVe vectors [Pennington et al., 2014].

Training

Given the reference captions, the proposed approach is trained by maximizing the log-likelihood function:

here Θ stands for all the training parameters in our approach. Besides, N is the size of training set, gk t stands for the tth word in the reference caption of training video k, and T denotes the total time steps of the decoder LSTM. Following existing approaches [Venugopalan et al., 2015a], T is fixed as 80 in this paper, which is bigger than the longest caption.

Evaluation Metrices

In this paper, four metrices are employed to evaluate the quality of generated captions, they are BLEU [Papineni et al., 2002], ROUGE-L [Lin and Och, 2004], CIDEr [Vedantam et al., 2015], METEOR [Denkowski and Lavie, 2014], where BLEU has four versions, i.e., BLEU 1-4. In fact, most of them are originally proposed for machine translation. Recently, they are also widely used in video captioning, since the evaluation of the two tasks are both the comparison between generated sentence and reference sentence. Practically, in video captioning, the higher scores of these metrices indicate the higher similarity between the generated caption and the reference caption, i.e., the higher accuracy of the generated caption.

3.2 Results and Discussion

Results on the MSVD Dataset

Table 1 shows the performance of various approaches on the MSVD dataset. It should be noted that, for a fair comparison, the results listed here are all generated with the baseline of static frame feature extractors. Concretely, HRNE are with GoogLeNet [Szegedy et al., 2015], and others are with VGG-16 [Simonyan and Zisserman, 2014]. Besides, to our approach, the Faster-RCNNmodel employed in this part is on the basis of VGG-16, trained on MicroSoft COCO dataset.

In Table 1, all the compared approaches except FGM are based on RNN. Specifically, FGM is a template-based approach, which generates video captions based on a sentence template. It is a successful combination of visual feature and language statics, and has won the state-of-the-art performance in non-RNN based approaches. However, benefiting from the great ability of RNN in modeling sequence, it can be observed from Table 1 that RNN-based approaches get much better performance. Actually, the biggest difference between our approach and other RNN-based approaches lies in the encoding of visual features. Detailedly, Mean Pool encodes the visual feature by simply average pooling the frame features.

SA and p-RNN are attention based approaches, which generate visual features by the weighted sum of frame features or region features. However, in the above three approaches, the temporal information of video features is missed. Our approach performs better than them because the temporal information is captured by the encoder LSTM. Similarly, S2VT also employs LSTM to encode the frame features. But they are conditioned on global frame features. The better results of our approach have verified the effectiveness of object-aware tube features. Moreover, HRNE, HBA and MAM-RNN design hierarchical architectures to model the frame sequence, which increase the model complexity. Fortunately, our approach gets comparable results with them. Besides, our approach also performs better than LSTM-E which jointly exploits the visual and semantic information by integrating a visual-semantic embedding space into LSTM. But its visual input to LSTM is just the average pooling of frame features, which restricts the performance.

In Table 2, the influence of different features is analyzed. Here, all the compared approaches are with C3D (3D convolutional neural network) [Tran et al., 2015] or flow frames (generated by optical flow) [Venugopalan et al., 2015a] to exploit the dynamic information of the video. Moreover, some of them are even fused with static frame features. Meanwhile, to our approach, four versions of feature extractor is provided, i.e., Faster-RCNN based on different CNNs and trained on different datasets. It can be observed that even though our approach is not equipped with 3D convolutional network, all the four results are better than compared approaches. It is because the dynamic information is already captured by the encoder LSTM. Therefore, the results have shown the great advantages of our tube features. Besides, our approach has shown stable performance on the four feature extractors. Precisely, the results on Resnet-101 is slightly higher than VGG16. It is because Resnet-101 is much deeper, which indicates stronger capability to exploit the visual information. Additionally, there is little difference between the results on PASCAL VOC and Microsoft COCO. Actually, Microsoft COCO contains 80 categories of objects, where most of the objects appearing in the video are included. However, PASCAL VOC just has 20 categories, which means the Faster-RCNN trained on this dataset may not capable of covering the diversity of the objects in the video. But, it can be observed that there is little decline in the results on PASCAL VOC. It is because that the detected object regions not only capture the information of an exact object, but also the common objectness. Besides, only the region feature is utilized to form the tube, their category label is not considered. Therefore, as long as the object regions are detected, whether they are classified accurately is not important to our approach.

In Table 3, to analyze the influence of different components, we test several variants of our approach. 1) The frame tube is deleted from the tube set, i.e., just object tubes are considered. It can be observed that the performance declined significantly. It indicates that the background information also makes contributions to the accuracy of video captions, and the global frame feature is as important as object-aware features in video captioning. 2) The attention model is dropped, such that the visual feature at each step is generated by average pooling the tube features. The worse performance verified the necessity of emphasizing the most correlated tube in the caption generation. 3) The bi-directional LSTM encoder is replaced with a single LSTM, so that only the forward dynamic information is considered. The worse results compared with our full model illustrate the advantages of bi-directional LSTM in dynamic information capturing. Overall, the presented results in Table 3 have verified the necessity of frametube, attention model and the bi-directional LSTM encoder of our approach.

Besides, in Figure 3, we provide some examples of our results on the MSVD dataset. Generally, most of the captions generated by our approach can describe the video content accurately. Moreover, when generating the key words, our approach can automatically attend to the most correlated tubes. Precisely, it can be observed from Figure 3 that there are duplicate tubes in the third row. It is caused by the wrong split of our tube generation method. This phenomenon occurs sometimes, especially when frames change rapidly. Fortunately, our attention model can reduce the interference caused by this mistake. Overall, in Figure 3, we can draw the conclusion that with the tube features, our approach shows great advantages in video captioning.

Results on the Charades Dataset

Table 4 presents the results on the Charades dataset. As aforementioned, it is a more challenging dataset, so the performance reflected on the evaluation metrics is much lower than that on the MSVD dataset. In Table 4, the results of four compared approaches are presented, where S2VT, SA and MAM-RNN have been introduced before, and MAAM designs a memory-augmented attention model to adaptively focus on a subset of frames when generating the caption. Compared to traditional attention model, like SA, the main novelty of MAAM is that the attention weight is determined jointly by the current visual feature and the memories of past attention. Practically, the better performance of MAAM than SA has shown its superiority. However, most of the four compared approaches neglect the object features. Therefore, our approach with tube features can get better performance than them. Besides, it can be observed from Table 4 that our approach with four different feature extractors gets comparable results, which indicates the stability of our approach.

Ours: A person is grabbing something out of a window on the stairs. Ref: A person wearing a blanket and holding a peper on stairs.

Ours: A person looks into a bedroom and sits on a bed, they take a shirt off

and then eating a sandwich.

Ours: A person stands in a doorway picking up a drink of coffee. Ref: A person is walking up the stairs holding a towel.

Figure 4: Example results of our approach on the Charades dataset, where each video is represented by four frames and the captions are displayed below. Specifically, the sentence in red denotes the caption generated by our approach, and the sentence in blue represents the reference caption annotated by human beings.

In Figure 4, we present some examples of generated captions on Charades dataset. As aforementioned, Charades is a much challenging dataset. The primary reason is that its reference captions are much longer. From the reference captions and the frames displayed in Figure 4, it can be observed that some generated captions can basically describe the video content, but the accuracy is much lower than that on the MSVD dataset. Worse still, there are even some mistakes on the key words in the sentence, i.e., subjective, verb and objective. It is because there are a lot of small objects in the reference captions, such as phone, shoes, pillow and so on. Unfortunately, it is hard to capture the visual information of these small objects for both frame features and region features. In fact, it is a common problem for existing approaches. In the future, we plan to improve our approach to address this problem.

4 Conclusion

In this paper, we propose to generate video captions with tube features. Practically, it is an object-aware feature which captures the trajectories of objects in the video. In our approach, the Faster-RCNN is employed to first extract region proposals, and the regions belonging to the same objects are linked to be tubes. Then, a video caption generator is built following the encoder-decoder paradigm. Specifically, a bi-directional LSTM is employed to encoding both the forward and backward dynamic information of tubes, and a single LSTM extended with a tube-based attention model is utilized as the decoder to generate the caption word by word.