Multi-Turn Video Question Answering via Multi-Stream Hierarchical Attention Context Network

1 Introduction

Visual question answering is the visual information delivery mechanism that enables users to issue their queries and then collect the answers from the referenced visual contents. Multi-turn video question answering is a challenging task in visual question answering, which automatically generates the accurate answer according to the newly given question and Conversation Context conversation context. Currently, most of the existing visual question answering approaches mainly focus on the problem of single-turn video question answering [Zeng et al., 2017; Zhu et al., 2015; Mazaheri et al., 2016; Tapaswi et al., 2016; Zhao et al., 2017; Jang et al., 2017]. Although the existing proposed methods have achieved promising performance in the single-turn video question answering task, they may still be ineffectively extended to the problem of multi-turn video question answering, due to the lack of modeling the visual conversation context for answer inference.

∗Corresponding author is Jun Xiao.

Figure 1: Multi-Turn Video Question Answering.

In conversational video question answering task, the context information is particularly important to video question understanding, due to the casual and short video question content. We illustrate a simple example of conversational video question answering in Figure 1. We show that in order to generate the right answer for the question "where did the woman place them on?", the collective conversation context information is required for the answer inference. Thus, the simple extension of the existing single-turn video question answering methods is difficult to provide the satisfactory results. Unlike the single-turn video question answering, the multi-turn video question answering method generates the answer from the referenced video content according to the given question as well as the conversation context. The historical conversation context is often in a hierarchical structure and has two levels of sequential relationships, which are the words in conversation turn and conversation turns in the context. Furthermore, not all the conversation context information are equally important for multi-turn video question answering. Therefore, in order to achieve high-quality multiturn video question answering, it is important to model the hierarchical sequential relationships among conversation context and to identify the important contextual information for multi-turn video question answering.

In this paper, we study the problem of multi-turn video question answering from the viewpoint of hierarchical attention context network learning. We first propose the hierarchical recurrent neural networks with attention mechanisms to model the sequential relationships among conversation context as well as the importance of contextual information for context-aware question understanding. We then devise the multi-stream hierarchical neural networks with saptio-temporal attention mechanisms to learn the joint representation of video contents and context-aware question embedding. We next develop the hierarchical attention context network learning method with multi-step reasoning process for multi-turn video question answering, named as MHACN. When a certain question is given, MHACN can generate the answer for it based on the referenced video contents and its conversation context. The main contributions of this paper are as follows:

• Unlike the previous studies, we present the problem of multi-turn video question answering from the viewpoint of hierarchical attention context network learning. We propose the multi-stream hierarchical attention context network that learns the joint representation of dynamic video content according to the context-aware question understanding.
• We incorporate the multi-step reasoning process for the proposed multi-stream hierarchical attention context network to enable the progressive joint representation learning of the multi-stream attentional video and context-aware question embedding, which further improves the performance of multi-turn video question answering.
• We construct two large-scale datasets for multi-turn video question answering and validate the effectiveness of our proposed method through extensive experiments.

2 Multi-Turn Video Question Answering via Attention Context Networks

2.1 ProblemFormulation

Before presenting our method, we first introduce some basic notions and terminologies. We denote the video by v ∈ V and conversation context by u ∈ U, respectively. The frame-level representation for video v by v(f) = (v(f ) 1 , v(f) 2 , . . . , v(f) T (f) ), where T (f) is the number of frames in video v. The vi = {v(f) i1 , v(f) i2 , . . . , v(f) iK } is the set of region features in the i-th frame by pre-trained 2D-ConvNet. The segment-level representation of video v is denoted by v(s) = (v(s) 1 , v(s) 2 , . . . , v(s) T (s) ), where T (s) is the number of segments in video v and v(s) j is the embedding of the j-th segment by pre-trained 3D-ConvNet. We denote the conversation context u ∈ U by u = (u1, u2, . . . , uM), where uk is the k-th round question answering conversation. We then denote the question by q ∈ Q and the answer by a ∈ A, and the k-th round conversation uk is composed of question qk and answer ak.

Since the video representations and conversation context are sequential data with variant length, it is natural to choose the variant recurrent neural network called long-short term memory network (LSTM) [Hochreiter and Schmidhuber, 1997] to learn their feature representations. We first denote the output states of frame-level video representations using LSTM by h(f) = (h(f) 1 , h(f) 2 , . . . , h(f) T (f) ), where h(f) i is the output state of the i-th frame in video v. We then consider the output states of segment-level video representations by h(s) = (h(s) 1 , h(s) 2 , . . . , h(s) T (s) ), where h(s) j is the output state of the j-th segment in video v. We next denote the output state of question representation by h(q) and the output state of answer representation by h(a) , respectively.

Using the notations above, the problem of multi-turn video question answering is formulated as follows. Given the set of videos V , conversation context U, questions Q and the associated answers A, our goal is to learn the multi-stream hierarchical attention context network such that when a new question is issued, MHACN can generate the answer for it based on the referenced video content and current visual conversation context.

2.2 Multi-StreamHierarchical Attention Context Network Learning

We first propose the context-aware question understanding method to learn the coherent question representation with conversation context. We consider that the conversation context is in a hierarchical structure and has two levels of sequential relations among questions, answers and each round of conversation context within the structure. Furthermore, we note that not all parts of conversation context are equally important for question understanding. Therefore, we propose the hierarchical recurrent neural networks with fusion mechanisms to model the conversation context and then devise the attention-over-context mechanism to learn the context-aware question representation.

We employ the LSTM networks to learn the representation of the question and the answer in the k-th round of the conversation context, denoted by h(q) k and h(a) k . We then employ the joint representation of question-answer pair mechanism [Zhou et al., 2015], to learn the representation of the k-th round of the conversation context uk by fusing the output states of question h(q) k and answer h(a) k , given by

where + denotes the element-wise addition for the joint representation of the question and answer contents (i.e., h(q) k and h(a) k ). The projection matrix W(q) and W(a) are used for the fusion of question and answer representations. We consider that the g(·) is the element-wise scaled hyperbolic tangent function, which has shown the good performance for multimodal representation fusion in [Orr and M¨uller, 2003]. We then learn the representation of conversation context using LSTM networks based on the joint representations u = (u1, u2, . . . , uM), denoted by h(u) = (h(u) 1 , h(u) 2 , . . . , h(u) M ). We next learn the context-aware question representation with attention-over-context mechanisms. Given the input question q and the representation of conversation context h(u) = (h(u) 1 , h(u) 2 , . . . , h(u) M ), the attention-over-context score s(q,u) i is given by

where h(q) is the output state of question q using LSTM networks. The W(q) , W(u) are parameter matrices and b(q,u) s is the bias vector. The w(q,u) is the parameter vector for computing the attention-over-context score. For each round of conversation context ui, its activation for the given question q by the softmax function is given by α(q,u) i =

exp(s(q,u) i ) ∑i exp(s(q,u) i ) , which is the normalization of the attentionover-context scores. Thus, the conversation context attended question representation is given by h(q,u) = ∑i α(q,u) i h(u) i . Therefore, the context-aware question representation is given by ˆh(q) = h(q) + h(q,u) .

Given the context-aware question representation ˆh(q) , we first develop the hierarchical spatio-temporal attention networks to learn the frame-level question-aware video representation. Since the global representation of the frame may fail to capture all necessary information for answering the question [Li and Jia, ], it is natural to choose the spatial attention mechanism to automatically localize the targeted regions in each frame according to the question. Following the existing spatial attention mechanism [Li and Jia, ], we employ the object generator to produce a set of candidate regions that are most likely to be an object. We extract the frame-level feature using 2D-ConvNet [Krizhevsky et al., ] by v(f) = (v(f) 1 , v(f) 2 , . . . , v(f) T (f) ), where v(f) i = {v(f ) i1 , v(f) i2 , . . . , v(f) iK } is the set of region features of the i-th frame. The v(f) i1 , v(f) i2 , . . . , v(f) i(K-1) are the candidate region features and v(f) iK is the whole frame feature. Given the region feature of the i-th frame v(f) ij ∈ v(f) i with context-aware question representation ˆh(q) , its spatial attention score s(q,r) ij is given by

where ˆW(q) , W(r) are parameter matrices and b(q,r) s is the bias vector. The w(q,r) is the parameter vector for computing the frame-level spatial attention score. For each region feature, its activation for the given context-aware question representation ˆh(q) is given by α(q,r) ij = exp(s(q,r) ij ) ∑j exp(s(q,r) ij ) , which is the normalization of the spatial attention score. The spatially attended frame representation is given by ˆv(f) i = ∑j α(q,r) ij v(f) ij .

On the other hand, a number of frames in the video are redundant and irrelevant to the question. Thus, it is important to localize the relevant frames with the targeted information according to the question. We thus introduce the temporal attention mechanism to estimate the relevance of video frames according to the question. Given the spatially attended frames ˆv(f) = (ˆv(f) 1 , ˆv(f) 2 , . . . , ˆv(f) T (f) ), we first learn their latent state representations from LSTM networks by h(f) = (h(f) 1 , h(f) 2 , . . . , h(f) T (f) ). Then, for each frame h(f) i , its temporal attention score s(q,f) i is given by

where ˆW(q) , W(f) are parameter matrices and b(q,f) s is the bias vector. The w(q,f) is the parameter vector for computing the frame-level temporal attention score. For each frame, its activation for the given context-aware question representation ˆh(q) is given by α(q,f) i = exp(s(q,f) i ) ∑i exp(s(q,f) i ) , which is the normalization of temporal attention score. Thus, the temporally attended frame representation is given by ˆh(f) = ∑i α(q,f) i h(f) i .

We then develop the temporal attention networks to learn the segment-level question-aware video representation. We extract the segment-level feature using 3D-ConvNet [Tran et al., 2015] by v(f) = (v(s) 1 , v(s) 2 , . . . , v(s) T (s) ), and learn their latent state representations using LSTM by h(f) = (h(s) 1 , h(s) 2 , . . . , h(s) T (s) ). For each video segment h(s) i , its temporal attention score based on the context-aware question representation ˆh(q) is given by

where ˆW(q) , W(s) are parameter matrices and b(q,s) s is the bias vector. The w(q,s) is the parameter vector. For each video segment, its activation for the given contextaware question representation ˆh(q) is given by α(q,s) i =

exp(s(q,s) i ) ∑i exp(s(q,s) i ) . Thus, the temporally attended segment representation is given by ˆh(s) = ∑i α(q,s) i h(s) i .

Therefore, we learn the question-aware video representation using multi-stream hierarchical attention context network by yh(q)(u, v) = ˆh(f) ⊗ ˆh(s) , where ⊗ is the element-wise product operator. We then incorporate the multi-step reasoning process [Sukhbaatar et al., ] for the proposed multi-stream hierarchical attention context network to further improve the performance of multi-turn video question answering. Given the multi-stream hierarchical attention context network y(·), video v and conversation context u, the multi-stream hierarchical attention context network learning with multi-step reasoning process is given by

which is recursively updated. The question-aware video representation is returned after the k-th update, denoted by zk. The learning process of reasoning multi-stream hierarchical attention context networks is illustrated in Figure 2.

Following the existing visual question answering models [Antol et al., 2015; Kim et al., ; Li and Jia, ], we model the problem of multi-turn video question answering as a classification task with pre-defined classes. Given the questionaware video representation z, a softmax function is employed to classify z into one of the possible answers as

whereW(z) is the parameter matrix and b(z) a is the bias vector. We note that instead of using softmax function for answer prediction, it is also possible to utilize LSTM, taking the question-aware video representation z as input, to generate the free-form answers for the open-ended multi-turn video question answering.

3 Experiments

In this section, we first introduce two conversational video question answering datasets, and then conduct several experiments on them, to show the effectiveness of our approach MHACN for multi-turn video question answering.

3.1 Data Preparation

We construct the conversational video question answering datasets from YouTubeClips [Chen and Dolan, 2011] and TACoS-MultiLevel [Rohrbach et al., ]. The YouTubeClips data consists of 1,987 videos and TACoS-MultiLevel data is composed of 1,303 videos. Each YouTubeClips video is composed of 60 frames and each TACoS-MultiLevel video consists of 80 frames. For each video, five pairs of crowdsourcing workers from a professional company were invited to construct five different conversational video dialogs. In total, we have 37,228 video question answering pairs for TACoS-MultiLevel data and 66,806 ones for YouTubeClips data. And most of video dialogs in TACoS-MultiLevel data have five turns of conversation question answering pairs. We take 90% of constructed conversational video dialogs as the training data, 5% as the validation data and 5% as the testing ones. For each question, we compute the semantic similarity between its ground-truth answer and all other answers based on the Euclidean distance with the pre-trained glove embedding [Pennington et al., ], and then rank the top 50 answers as the candidate answer set. The constructed conversational video question answering datasets will be provided later.

We then process the conversational video question answering datasets as follows. We resize each frame to 224 × 224 and extract the visual representation of each frame by the pretrained VGGNet [Simonyan and Zisserman, 2014], and take the 4,096-dimensional feature vector for each frame. We employ the pre-trained word2vec model [Mikolov et al., 2013] to extract the semantic representation of questions and answers. Specifically, the size of vocabulary set is 6,500 and the dimension of word vector is set to 256. We evaluate the performance of our proposed MHACN method based on three widelyused ranking evaluation criteria MRR, P@K and MeanRank for multi-turn video question answering, which has been widely used in visual question answering [Zeng et al., 2017; Jang et al., 2017; Zhao et al., 2017].

3.2 Performance Comparisons

We employ the existing single-turn video question answering algorithm and extend the existing video question answering methods as the baseline algorithms for the problem of multiturn video question answering.

• ESA method is the single-turn video question answering algorithm [Zeng et al., 2017], which learns the joint video representation based on the given question with attention mechanisms.
• ESA+ method is the extension of ESA algorithm [Zeng et al., 2017], where we add the LSTM network to model the conversation context and then fuse context representation and question embedding into the joint representation for multi-turn video question answering.
• STVQA+ method is the extension of STVQA algorithm [Jang et al., 2017], where we add the LSTM network to sequentially fuse the conversation context and spatial-temporal attended video for multi-turn video question answering.
• STAN+ method is the extension of STAN algorithm [Zhao et al., 2017], where we add the LSTM network for context-aware question understanding and then perform spatio-temporal attention with context-aware question representation for multi-turn video question answering.

Unlike the previous video question answering works, our MHACN method performs the context-aware question understanding and learn multi-stream spatio-temporal attention video representation with multiple reasoning process for the problem. To exploit the effect of multi-stream hierarchical attention process, we denote the spatio-temporal hierarchical attention context network with 2D-ConvNets by SHACN, and the temporal hierarchical attention context network with 3D-ConvNets by DHACN. Next, to validate the effect of reasoning process, we denote that our method without reasoning process by MHACN, and our method with reasoning steps by MHACN(r). The input words of out method are initialized by pre-trained word embeddings [Mikolov et al., 2013] with size of 256, and the weights of LSTM networks are randomly by a Gaussian distribution with zero mean.

Table 1 shows the experimental results of the methods on MRR, P@1, P@5 and MeanRank using TACoS-MultiLevel dataset. Table 2 demonstrates the evaluation results of the methods using YoutubeClip dataset. The hyperparameters and parameters which achieve the best performance on the validation set are chosen to conduct the testing evaluation.

We report the average value of all the methods on three evaluation criteria. The experimental results reveal a number of interesting points:

• The methods based on context-aware question understanding, ESA+, STVQA+, STAN+, SHACN, DHACN, MHACN and MHACN(1) outperform the single-turn video question answering method ESA, which suggests the context-aware question representation is critical for the problem.
• The multi-stream attention context network method MHACN achieves better performance than the methods SHACN and DHACN. This suggests that both the frame-level and segment-level hierarchical attention mechanisms are important for the problem of multi-turn video question answering.
• In all the cases, our MHACN(1) method achieves the best performance. This fact shows that the reasoning multi-stream attention context network learning framework that exploits both context-aware question understanding and multi-stream attention mechanisms, and multiple reasoning updates can further improve the performance of multi-turn video question answering.

In our approach, there are two essential parameters, which are the dimension of question representation, and the dimension of the joint representation of the question-answer pairs. We vary the dimension of question representation from 32, 60, . . ., to 512, and the dimension of joint representation from 32, 64, . . ., to 512. We first investigate the effect of question representation dimension on MRR, P@1 and P@5 on TACoS-MultiLevel dataset in Figures 3, and on YoutubeClip dataset in Figures 4. We then study the effect of joint representation dimension on TACoS-MultiLevel dataset in Figures 5.

4 RelatedWork

In this section, we briefly review some related work on visual question answering and dialogue modeling.

The visual question answering task is to provide the accurate answer for the natural language question from the given visual contents [Antol et al., 2015]. The existing approaches can be categorized into image-based question answering methods [Antol et al., 2015; Li and Jia, ; Yang et al., 2016] and video-based question answering ones [Mazaheri et al., 2016; Zeng et al., 2017; Zhu et al., 2015; Zhao et al., 2017; Tapaswi et al., 2016; Jang et al., 2017]. Kim et. al. [Kim et al., ] employ the multimodal residual network for image question answering. Li et. al. [Li and Jia, ] propose the question representation update method that iteratively selects the relevant image regions and update the question representation. Shih et. al. [Shih et al., 2016] introduce the spatial attention mechanism for image question answering. Das et. al. [Das et al., 2017] study the image question answering based on previous questionanswering history. As a natural extension of image-based question answering, the video-based question answering has been introduced as a more challenging task [Zeng et al., 2017]. The fill-in-the-blank approaches [Zhu et al., 2015; Mazaheri et al., 2016] complete the missing entry in the video description by ranking candidate answers based on both visual content and contextual video description. Tapaswi et. al. [Tapaswi et al., 2016] propose the three-way scoring function for movie question answering Zhao et. al. [Zhao et al., 2017] propose the hierarchical spatio-temporal attention networks for video question answering. Jang et. al. [Jang et al., 2017] devise the dual-LSTM method with attention mechanism. Unlike the previous studies, we study the problem of conversational video question answering based on both the visual contents and its conversational context.

Given a dialogue context in natural language, the response generation task is to provide the relevant utterance to the given conversational context [Serban et al., 2017b]. Serban et. al. [Serban et al., 2016] extend the hierarchical recurrent encoder-decoder neural network for responding learning in dialogue systems. Weston et. al. [Weston, 2016] study the dialog-based language learning based on memory network. Serban et. al. [Serban et al., 2017a] devise the multiresolution neural network model for dialogue response generation. Mei et. al. [Mei et al., 2017] study the coherent conversation continuation via RNN-based dialogue models with dynamic attention mechanisms. Unlike the previous studies, the conversational video question answering task is to provide the answer from the multimodal visual contents and textual conversational contexts.

5 Conclusion

In this paper, we study the problem of multi-turn video question answering from the viewpoint of multi-step hierarchical attention context network learning. We first propose the hierarchical attention context learning method with recurrent neural networks for context-aware question understanding, which is based on the sequential conversation context structure with attention-over-context mechanisms. We then develop the multi-stream attention network that learns the joint embedding for video question answering from both the spatio-temporal attended frame-level video representation and the temporal attended segment-level video representation. We next incorporate the multi-step reasoning process for the proposed multi-stream hierarchical attention context network to further improve the performance of multi-turn video question answering. We construct two large-scale multi-turn video question answering datasets and evaluate the effectiveness of our proposed method through extensive experiments.

Acknowledgements

This work was supported by National Natural Science Foundation of China under Grant No.61602405, No.61572431, Zhejiang Natural Science Foundation(LZ17F020001), Key R&D Program of Zhejiang Province(2018C01006) and Joint Research Program of ZJU and Hikvision Research Institute.