Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks

1 Introduction

Open-ended video question answering is an important problem in visual information retrieval, which automatically generates the accurate answer from referenced video contents according to the given question. Most existing works employ the multi-modal recurrent encoder-decoder framework, which first encodes the multi-modal video and question contents into a joint representation, and then generates the natural-language answer [Xue et al., 2017; Zhao et al., 2018]. Although these approaches have achieved excellent performance in short-form video question answering, they may still be ineffectively applied to the long-form video question answering due to the lack of long-range dependency modeling and the suffering from the heavy computational cost.

∗Zhou Zhao is the corresponding author.

Figure 1: Open-Ended Long-Form Video Question Answering.

The long-form video often contains the complex information of the targeted objects that evolve over time [Krishna et al., 2017]. We illustrate a concrete example in Figure 1. The answer generation to the question "What does the girl do after putting the fruit into a blender?" requires the accurate information extraction from long-form video contents. However, the sequential processing of long-form video contents by recurrent neural networks remains the network depth-in-length, which causes the heavy computational cost and insufficiency of long-range dependency modeling. Therefore, the simple extension of multi-modal recurrent encoder-decoder networks is difficult to generate a satisfactory answer for openended long-form video question answering.

Recently, the convolutional sequence-to-sequence method has been applied in fast conditional text generation [Gehring et al., 2017]. Based on the convolutional sequence modeling, we are able to improve the computational efficiency of long-form video encoding. However, the original convolutional sequence-to-sequence method still has some limitations. Firstly, the temporal convolution keeps the sequential length unchanged, so the multi-layer convolution operation leads to the large memory requirement, especially for longform video sequences. Secondly, as shown in Figure 1, the crucial clues for question answering often are contained in continuous segments rather than discrete frames, but the original convolutional sequence-to-sequence strategy ignores the hierarchical structure of long-form video contents. Thirdly, this method still remains in local modeling and fails to capture long-range dependencies from video context.

We tackle these problems by a hierarchical convolution self-attention encoder, which can efficiently learn multi-layer video semantic representations. Specifically, the encoder consists of multiple convolution self-attention layers, and each layer contains two convolution units, an attentive segmentation unit and a question-aware self-attention unit. The attentive segmentation unit splits long-form video contents into different segments and leverages question information to learn attentive segment-level representations, which builds the hierarchical structure of long-form video contents and explores multi-scale visual clues for question answering. Moreover, this method reduces the video sequential length layer by layer and is helpful to alleviate memory load. On the other hand, the question-aware self-attention unit exploits the question evidence to capture long-range dependencies from video context. Concretely, different from the conventional self-attention developed for machine translation [Vaswani et al., 2017], we consider that a large portion of contents in long-form videos are irrelevant to the given question and even interfere with question answering, thus utilize the question information as guidance for filtering the unnecessary context.

After encoding long-form video contents, we consider the decoder network for answer generation. Previous works [Bahdanau et al., 2014; Anderson et al., 2018] often adopt the recurrent neural network with attention mechanism to generate the natural-language answer. Different from them, we develop a multi-scale attentive recurrent decoder to incorporate multi-layer semantic representations from the hierarchical video encoder according to the given question, which avoids the information missing of the top layer.

In summary, the main contributions of this paper are as follows:

• we develop a hierarchical convolutional self-attention encoder to efficiently explore the hierarchical structure of long-form video contents and capture question-aware long-range dependencies from video context.
• We devise a multi-scale attentive decoder to incorporate multi-layer video semantic representations for answer generation, which avoids information missing only from the top layer of the hierarchical encoder.
• The extensive experiments on a public long-form video question answering dataset validate the effectiveness and efficiency of our method.

2 RelatedWork

In this section, we briefly review some related work on visual question answering.

Given a question to an image, the image-based question answering task is to return the accurate answer for the given question [Antol et al., 2015]. [Li and Jia, 2016] propose the QRU method that updates the question representation iteratively by selecting image regions. [Malinowski and Fritz, 2014] combine discrete reasoning with uncertain predictions in a bayesian framework for automatically answering questions. [Lu et al., 2016] devise the co-attention mechanism for image question answering that generates spatial maps. A detailed survey of existing image question answering approaches can be found in [Wu et al., 2017].

As a natural extension, the video-based question answering has been proposed as a more challenging task [Tapaswi et al.,

2016; Jang et al., 2017]. The fill-in-the-blank methods [Maharaj et al., 2017; Mazaheri et al., 2016] complete the missing entries in the video description based on both visual and textual contents. [Zhu et al., 2017] present the encoder-decoder approach to learn temporal structures of videos. [Gao and Han, 2017] encode the video by LSTM and then decode the answer by a language model. [Xue et al., 2017] apply the sequential video attention and temporal question attention for open-ended video question answering. [Jang et al., 2017] propose the dual-LSTM based approach to perform spatiotemporal reasoning from videos to answer questions. [Zeng et al., 2017] extend the end-to-end memory network with additional LSTM layer for video question answering. [Zhao et al., 2018] adaptively divide entire videos into several segments to model video presentation for answer generation. [Gao et al., 2018] devise the dynamic memory network to learn temporal video representations for question answering. Besides singleturn video question answering, [Zhao et al., 2019] propose a hierarchical attention method to solve the multi-turn video question answering task.

Although these methods have achieved promising performance, they may still suffer from the heavy computational cost and the ineffectiveness of long-range dependency modeling for long-form video contents.

3 Hierarchical Convolutional Self-Attention Encoder-Decoder Networks

3.1 The Problem

We present a video as a sequence of frames v = {vi}ni=1 ∈ V , where vi is the feature of the i-th frame and n is the feature number. Each video is associated with a natural language question, denoted by q = {qi}mi=1 ∈ Q, where qi is the feature of the i-th word and m is the word number. And the ground-truth answer for each question is denoted by a = {ai}ri=1 ∈ A of length r, where ai is the i-th word token. Our goal is to generate a natural-language answer from referenced long-form video contents according to the given question. The overall framework is shown in Figure 2.

3.2 Hierarchical Convolutional Self-Attention Encoder

In this section, we introduce the hierarchical convolutional self-attention encoder that unifies the convolutional sequence modeling, attentive segmentation and question-aware selfattention mechanism into a common framework.

We first extract the visual features v = {vi}ni=1 using a pre-trained 3D-ConvNet [Tran et al., 2015] and then apply a linear projection for dimensionality reduction. Compared with RNN-based encoder, convolutional encoder lacks precise position modeling. Hence, we add position encoding [Vaswani et al., 2017] into the initial video sequence to supplement the temporal information. After that, we employ a pre-trained word2vec [Mikolov et al., 2013] to extract the word features q = {qi}mi=1 , and then develop a bi-directional GRU networks (BiGRU) to learn the question semantic representations. The BiGRU networks incorporate contextual information for each word, given by

where GRUfq and GRUbq represent the forward and backward GRU, respectively. And the contextual representation hqi is the concatenation of the forward and backward hidden state at the i-th step. Thus, we get the question semantic representations hq = (hq1 , hq2 , . . . , hqm) and global question represenattion hQ.

Next, we introduce the hierarchical convolutional selfattention encoder, composed by L convolutional selfattention layers. The l-th layer takes the sequence representations hl-1 = (hl-1 1 , hl-1 2 . . . , hl-1 nl-1 ) of length nl-1 as input, and output another sequence representations hl = (hl1 , hl2 , . . . , hlnl ) of length nl, where nl-1 = H×nl and the sequence is reduced by a factor of 1/H per layer. The input of the first layer is the initial video sequence v = {vi}ni=1 with n elements. Concretely, each layer consists of two convolution units, an attentive segmentation unit and a questionaware self-attention unit.

Convolution Unit

The convolution unit can efficiently model the local context of long-form video contents. We first conduct a onedimensional convolution for the input sequence. Specifically, we parameterize the convolution kernel by Wl ∈ R2d×kd and bl ∈ R2d , where k is the kernel width and d represents the dimensionality of sequence elements. We then concatenate successive k elements and map them into a single output Y ∈ R2d with the convolutional kernel, given by Y = Wl [hl-1 i-k/2 ; . . . ; hl-1 i+k/2 ] + bl . We pad the input sequences by k/2 zero vectors on the both sides to keep the sequence length invariable.

We then devise a special non-linearity operation by the gated linear unit, called as GLU [Gehring et al., 2017], on the output Y = [A; B] ∈ R2d :

where A, B ∈ Rd are the inputs to the GLU, δ is the sigmoid function, ⊗ represents the point-wise multiplication and the output GLU([A,B]) ∈ Rd has the same dimensionality as the input element. Furthermore, to build deeper convolutional encoder, we add residual connection for each convolutional unit, given by

We note that ol1 i contains information over k input elements and the receptive field of final output can be expanded rapidly by stacking several units. And the output of the two convolutional units of the l-th layer is (ol2 1 , ol2 2 , · · · ; ol2 nl-1 ).

Attentive Segmentation Unit

The attentive segmentation unit splits long-form video contents into different segments and leverages question information to learn attentive segment-level representations. Thus, this method can reduces the video sequential length layer by layer to build the hierarchical video structure and also alleviate memory load.

Given the convolution output (ol2 1 , ol2 2 , · · · ; ol2 nl-1 ), we first devide sequence elements into nl segments, each containing H elements (i.e. nl-1 = H × nl). We then devise an attention mechanism to aggregate element representations for each segment, where we utilize the question representation as guidance to highlight critical elements. Different from mean-pooling aggregation, this method can effectively filter the irrelevant video contents for subsequent modeling.

Concretely, for the i-th segment (ol2 (i-1)H+1 , . . . , ol2 iH ) with H elements, we compute the attention weight of j-th element ol2 (i-1)H+j according to the global question representation hQ, given by

where W1 s , W2 s are parameter matrices, bs is the bias vector and the w⊤ s is the row vector for computing the attention weight. We then conduct the softmax operation on attention weights and obtain the segment representation sli by sli = ∑Hj=1 softmax(αij)ol2 (i-1)H+j .

Hence, by the attentive segmentation unit, we learn the segment-level video representations sl = (sl1 , sl2 . . . , slnl ). This segmentation strategy builds the hierarchical structure of long-form video contents, which is helpful for exploring multi-scale viusal clues for question answering. Moreover, this method reduces the video sequential length layer by layer to alleviate the memory load.

Question-Aware Self-Attention Unit

The question-aware self-attention unit captures the longrange dependencies from long-form video context with question information as guidance, which filters the unnecessary context.

Given the ouput of the attentive segmentation unit sl = (sl1 , sl2 , . . . , slnl ) and question semantic representations hq = (hq1 , hq2 , . . . , hqm), we can compute the video-to-question attention weights between each pair of sequence element and word, and obtain a video-to-question attention matrix M ∈ Rnl×m. Specifically, the attention weight of the i-th element and j-th word is calculated by

We then calculate the self-attention matrix D for video contents based on M as below:

where each value in D represents the correlation of two video elements. Specifically, each value in D is calculated by Dij = ∑mk=1 SikS⊤kj , where k represents the index of the k-th word in the question. That is, we regard question semantic representations as the middle layer while establishing element-to-element correlation of the video sequence. Compared with the conventional self-attention method, we filter the ineffective video context with question contents as guidance while capturing long-range dependencies.

We then conduct the softmax operation for each row in the self-attention matrix D and compute self-attention representations for each element, followed by an additive residual connection:

Therefore, we obtain the final output of the l-th convolutional self-attention layer hl = (hl1 , hl2 . . . , hlnl ).

By stacking L convolutional self-attention layers, we learn multi-layer video semantic representations (h1 , h2 , . . . , hL).

3.3 Multi-Scale Attentive Decoder Network

In this section, we introduce the multi-scale attentive decoder to consider the multi-scale visual clues from the hierarchical encoder for natural-language answer generation.

Our multi-scale attentive decoder is based on a GRU answer generator. At each time step, we conduct a recurrent operation as follows:

where xt is the input vector and hat is the output vector of the t-th step. The xt consists of the embedding wt of the input word at the t-th step, global question representation hQ and multi-scale video representation hvt ], given by xt = [wt; hQ; hvt ]. We then introduce how to develop the multiscale video representations hvt . By hierarchical encoder, the long-form video sequences are exponentially shorten layer by layer, and the output from the top layer of the hierarchical encoder aggregates informative segment-level representations for question answering. However, only considering the top-layer output may ignore fine-grained features in preceding layers and lead to information missing. Thus, after trade-off consideration, we devise a multi-scale attention on encoder outputs of top-K layers. Specifically, for the l-layer video representations hl = (hl1 , hl2 , . . . , hlnl ), we compute the attention weight of i-th element with the hidden state hat-1 and glabal question representation hQ, given by

We then conduct the softmax operation on attention weights and obtain the l-layer attentive representation vl by

Based on top-K attentive representations, we finally get the multi-scale video representation hvt by a mean pooling:

With xt as input, we get the output vector hat of GRU networks at the t-th time step, and further generate the conditional distribution over possible words, given by

where ˆat is the t-th generated word and T is the size of the vocabulary. Finally, we apply the maximum likelihood estimation to train our encoder-decoder networks in an end-to-end manner, given by

4 Experiments

4.1 Dataset

We conduct experiments on an open-ended long-form video question answering dataset [Zhao et al., 2018], which is constructed from the ActivityCaption dataset [Krishna et al., 2017] with natural-language descriptions. The average video time in the dataset is approximately 180 seconds and the longest video even lasts over 10 minutes. The questionanswer pairs contain five types corresponding to the object, number, color, location and action for the video contents. The first four types of questions mainly focus on the appearance visual features in videos but motion features are necessary to the action related question answering. The details of this dataset are summarized in Table 1.

4.2 Implementation Details

In this section, we introduce data preprocessing and model settings.

We first resize each frame to 112 × 112 and then employ the pre-trained 3D-ConvNet [Tran et al., 2015] to extract the 4,096-d feature for each unit, which contains 16 frames and overlaps 8 frames with adjacent units. We then reduce the dimensionality of the feature from 4,096 to 500 using PCA. Next, we set the maximum sequence length to 512 and downsample overlong sequences to this length. As for the question, we employ the pre-trained word2vec model [Mikolov et al., 2013] to extract the semantic features word by word. The dimension of word features is 300 and the vocabulary size T is 10000.

In our HCSA, we set the layer number L of the hierarchical convolutional self-attention encoder to 3. And the segmentation factor H in the attentive segmentation unit is set to 4. To avoid heavy computational cost, we only consider top-2 layers (K = 2) of the hierarchical encoder for the multi-scale attentive decoder. Moreover, we set convolution kernel width k to 5, convolution dimension to 256 and the dimension of the hidden state of GRU networks to 256 (512 for BiGRU while question encoding). And the dimensions of the linear matrice in all kinds of attention are set to 256. During training, we adopt an adam optimizer to minimize the loss and the learning rate is set to 0.001.

4.3 Performance Criteria

We evaluate the performance of open-ended video question answering based on evaluation criteria BLEU-1, WUPS@0.0 and WUPS@0.9. Since the length of answers is relatively short, we mainly focus on word-level evaluation. The BLEU1 is for accurate word matching and WUPS [Malinowski and Fritz, 2014] accounts for word-level ambiguities. Given the ground-truth answer a = {a1, a2, . . . , ar} and the generated answer ˆa = {ˆa1, ˆa2, . . . , ˆar}, the WUPS is given by

where the WUPγ(·) [Wu and Palmer, 1994] calculates word similarity by the WordNet [Fellbaum, 1998], given by

We set the threshold γ to 0 and 0.9 and denote the two criteria byWUPS@0.0 andWUPS@0.9, respectively.

4.4 Performance Comparisons

The open-ended video question answering is an emerging task, thus we compare our HCSA method with existing openended methods and meanwhile extend some conventional multi-choice methods for performance evaluation. Specifically, we add a GRU recurrent answer generator with attention mechanism to the end of those non-open-ended models.

• MN+ method [Zeng et al., 2017] is the extension of endto-end memory network algorithm, where we add a biLSTM network to encode the sequence of video frames.
• STVQA+ method [Jang et al., 2017] utilizes the spatial and temporal attention strategies on videos to answer related questions.
• CDMN+ method [Gao et al., 2018] proposes a motionappearance co-memory network to simultaneously learn the motion and appearance features.
• UNIFYmethod [Xue et al., 2017] applies the sequential video attention and temporal question attention for openended video question answering.
• AHN method [Zhao et al., 2018] divides entire videos into several segments and adopts a hierarchical attention to model video presentations for answer generation.

The former three approaches are originally developed for multi-choice video question answering and we extend them into the open-ended form. Table 2 shows the overall experimental results of all methods on three criteria BLEU-1, WUPS@0.9 and WUPS@0.0. Table 4 and 5 demonstrate the evaluation results of different question types on BLEU-1 and WUPS@0.9, respectively. Moreover, we adjust the parameter number in different methods at the same magnitude for fairly evaluating the time consumption, shown in Table 3. The training and inference time for each epoch only contain the network execution time. The experiment results reveal some interesting points:

• The methods based on attention mechanism, UNIFY, STVQA+, CDMN+, AHN, and HCSA achieve better

performance than MN+, which demonstrates that the joint representation learning of video and question is critical for high-quality answer generation.

• Our HCSA method achieves the best performance on three evaluation criteria and all question types, especially for action type. It suggests that our hierarchical convolutional self-attention encoder and the multi-scale attentive decoder are effective for the problem.
• From the perspective of time consumption, our HCSA method reduces plenty of training and testing time per epoch compared with all baselines, and can converge more quickly using quite a few epochs. This suggests that the convolution-based sequence modeling is more efficient than RNN-based sequence modeling.

4.5 Ablation Study and Visualization

To prove the contribution of each component of our HCSA method, we next conduct some ablation studies. Concretely, we discard or change one component at a time to generate an ablation model. We first replace the segment attention in the attentive segmentation unit with a conventional meanpooling, denoted by ASU(MP), and further remove attentive segmentation unit as w/o. ASU. We then replace the questionaware self-attention with a typical self-attention, denoted by QSU(SA), and thoroughly discard the question-aware selfattention unit as w/o. QSU. We finally denote the model that only considers the top-layer video representations in the decoder by w/o. MA. The ablation results are shown in Table 6.

By analyzing the ablation results, we find the full method outperforms w/o. ASU, w/o. QSU and w/o. MA models, indicating the attentive segmentation, question-aware self-attention and multi-scale attention are all helpful for high-quality answer generation. Additionally, the ASU(MP) and QSU(SA) still achieve worse performance than the full model, which demonstrates our proposed segment attention and question-aware self-attention is more effective than conventional mean-pooling and self-attention. Moreover, the worst performance of w/o. ASU suggests the hierarchical structure is crucial for long-form video modeling.

Furthermore, we demonstrate how the multi-scale attentive decoder works by a visualization example. As shown in Figure 3, we display the multi-scale attention results for video representations using a thermodynamic diagram, where the darker color represents the higher correlation. For each step of answer generation, the multi-scale attention produces a weight distribution over the video semantic representations of top-2 layers, We note that our proposed method can attend the semantically related visual contents and ignore these irrelevant features at each step. It suggests our proposed decoder effectively incorporates the multi-scale visual clues for highquality open-ended long-form video question answering.

5 Conclusion

In this paper, we propose a fast hierarchical convolutional self-attention encoder-decoder network for open-ended longform video question answering. We first propose a hierarchical convolutional self-attention encoder to efficiently model long-form video contents, which builds the hierarchical structure for video sequences and captures question-aware longrange dependencies from video context. We then devise a multi-scale attentive decoder to incorporate multi-layer video representations for natural-language answer generation. The extensive experiments show the effectiveness and efficiency of our method.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant No.61602405, No.U1611461, No.61751209 and No.61836002, Joint Research Program of ZJU and Hikvision Research Institute. This work is also supported by Alibaba Innovative Research and Microsoft Research Asia.