Video Question Answering via Hierarchical Spatio-Temporal Attention Networks

1 Introduction

Visual information retrieval (VIR) is the visual information delivery mechanism that enables users to post their queries and obtain the relevant information in visual media. Visual question answering is the essential problem in VIR sites, which automatically returns the relevant answer from the reference visual content according to the user's given question. Most of the existing works consider the problem of open-ended visual question answering as the multimodal understanding task, which learn the joint representation from the multimodal features of the given visual content and textual question, and then generate the natural language answer. However, the existing works mainly tackle the problem of static image question answering [Antol et al., 2015; Shih et al., 2016; Lu et al., 2016; Li and Jia, 2016; Xiong et al., 2016]. Although the existing works have achieved promising performance in image question answering, they may still be ineffectively applied to video question answering due to the lack of modeling the temporal dynamics of video content.

Figure 1: Open-ended Video Question Answering.

The video contents often contain the complex interactions of the targeted objects to the given question that evolves over time [Yao et al., 2015]. Thus, the simple extension of existing image question answering methods based on the single temporally collapsed video representation is likely to generate unsatisfactory answers. We illustrate a simple example of open-ended video question answering in Figure 1. We show that the answer generation to the question "what is the cat doing?" requires the collective information frommultiple video frames. Recently, temporal attention mechanism has been proposed to extract the critical frame information across the entire video for representation learning [Yao et al., 2015]. We then employ the temporal attention mechanism to model the collective information of video content for video question answering. On the other hand, the sequential order of the frames is also important for video representation [Fernando and Gould, 2016]. Thus, leveraging the sequential complex interactions of the targeted objects according to the question from the collective frame-level video representation is critical for modeling the temporal dynamics of video content in video question answering.

In this paper, we present the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We propose the hierarchical spatio-temporal attention networks that jointly learn the representation of the sequentially critical frames with the targeted objects according to the question. We then develop the encoder-decoder learning framework that enables the joint representation learning of the multimodal spatio-temporally attentional video and textual question through multiple reasoning steps for open-ended video question answering, named as r-STAN. When a certain question is issued, r-STAN can generate natural language answer for it based on the reference video content. The main contributions of this paper are as follows:

• Unlike the previous studies, we study the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We propose the spatio-temporal attention networks that learn the joint representation from the critical video frames of the targeted objects according to the question.
• We incorporate the multi-step reasoning process for the proposed spatio-temporal attention networks to enable the progressive joint representation learning of the multimodal spatio-temporal attentional video and textual question to further improve the performance of video question answering.
• We construct a large-scale dataset for open-ended video question answering and validate the effectiveness of our proposed method through extensive experiments.

2 Video Question Answering via Spatio-Temporal Attention Networks

In this section, we study the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We first develop the spatio-temporal attentional encoder networks with multi-step reasoning process to learn the joint representation of multimodal saptio-temporal attentional video and textual question progressively. We then devise the recurrent decoder network to generate the natural language answer for openended video question answering.

Before presenting the learning framework, we first introduce some basic notions and terminologies. We denote the question by q ∈ Q, the video by v ∈ V and the answer by a ∈ A, where Q, V and A are the sets of questions, videos and answers, respectively. Since the video is composed of sequential frames, the frame-level representation of video v is given by v = (v1, v2, . . . , vN) of length N, where v2 is the second frame. We then denote the word-level representation of natural language answer by a by a = (a1, a2, . . . , aM) of length M, where aM is the Mth word token. Since both the video and answer are sequential data with variant length, it is natural to choose the variant recurrent neural network called gated recurrent unit (GRU) [Chung et al., 2014] to learn the feature representation by:

where xt and ht are the input and output vectors, zt and rt are the update and reset gate vectors, Ws and bs are the parameter matrices and bias vector. We note that gate vector zt in Equation (2) is the trade-off parameter for updating hidden state ht from the previous state ht-1 and the currently estimated one ˜ht. Specifically, we learn the sequential feature representation of both video and answer by directional GRU, which consists of a forward GRU and a backward GRU. The backward GRU has the same network structure with the forward one while its input sequence is reversed. We denote the hidden state of the forward GRU for the video at time t by hft , and the hidden state of the backward GRU by hbt . Thus, the tth hidden state of video v from the bidirectional GRU layer is denoted by ht = [hft , hbt ], and the hidden states of video v is given by h = (h1, h2, . . . , hN).

Using the notations above, the problem of open-ended video question answering is formulated as follows. Given the set of videos V , questions Q and answers A, our goal is to learn the encoder-decoder network model g(f(v, q)) where the encoder network f(v, q) that learns the joint representation of the video and question, and the decoder network ˆa = g(f(v, q)) generates the answer ˆa for open-ended video question answering. We present the details of the spatiotemporal attention network learning framework in Figure 2.

2.1 Hierarchical Spatio-Temporal Attentional Encoder Network Learning

In this section, we propose the encoder neural network f(·) to learn the joint representation of video and question with hierarchical spatio-temporal attention and multiple reasoning updates for the problem of open-ended video question answering.

Inspired by attention mechanism [Xu et al., 2015], we propose the hierarchical spatio-temporal attention networks to learn the joint representation from the relevant frames of the targeted regions according to the question. Since the global representation of the frame may fail to capture all necessary information for answering the question [Li and Jia, 2016], it is natural to choose the spatial attention model to automatically localize the targeted regions in each frame according to the question. Following the existing spatial attention model [Li and Jia, 2016], we employ the object generator to produce a set of candidate regions that are most likely to be an object. The frame representation is thus given by the set of the candidate region features and the whole frame region feature, denoted by F = {F1, F2, . . . , FN}. The Fj = {fj1, fj2, . . . , fjK} is the feature set of the j-th frame, where fj1, fj2, . . . , fj(K-1) are the candidate region features and fK is the whole frame region feature. Given question q and the region feature of j-th frame representation fji ∈ Fj, the spatial attention score s(s) ji is given by

where Wqs and Wfs are parameter matrices and bs is bias vector. For each region fji, the activations in spatial dimension by the softmax function is given by αji = exp(s(s) ji ) ∑i exp(s(s) ji ) , which is the normalization of the spatial attention score. The spatially attended frame representation is then given by v(s) j = ∑i αjifji.

On the other hand, a number of frames in the video are redundant and irrelevant to the question. Thus, it is important to localize the relevant video frames with the targeted information according to the question. We thus introduce the temporal attention model to estimate the relevance of video frames according to the question. Given the spatially attended video frames v(s) = (v(s) 1 , v(s) 2 , . . . , v(s) N ), we learn their latent state representation from bidirectional GRU layer by h(s) = (h(s) 1 , h(s) 2 , . . . , h(s) N ). Then, their relevance scores based on temporal attention mechanism [Xu et al., 2015] is given by

where Wqt and Wht are parameter matrices and bt is bias vector. For the latent state of each frame h(s) j , its activation in temporal dimension by the softmax function is denoted by βj = exp(s(t) j ) ∑j exp(s(t) j ) , which is the normalization of

the temporal attention scores (i.e., βj ∈ (0, 1)). Inspired by the attentional gate [Kumar and Irsoy, 2015], we employ the attentional GRU networks to learn the order-sensitive representation of the spatio-temporally attended, denoted as aGRU network. The inputs to the aGRU network are the latent state of spatially attended frames by bidirectional GRU layer h(s) 1 , h(s) 2 , . . . , h(s) N , and the estimated temporal attention scores β1, β2, . . . , βN. The current estimated state ˜h(t) j is obtained based on the input h(s) j . The aGRU network then updates its hidden state h(t) j based on the mixture of current estimated state ˜h(t) j and its previous state h(t) j-1 . Unlike the update rule of GRU in Equation (4), the aGRU updates the current state h(t) j , given by

where the update gate vector is set to the normalized temporal attention score (i.e., zt = βt). Therefore, spatio-temporal attentional representation of video v according to question q is given by hsp q (v) = h(t) N , where h(t) N is the last hidden state of the aGRU networks.

We then incorporate the multi-step reasoning process [Sukhbaatar et al., 2015] for the proposed spatiotemporal attention networks to further improve the performance of open-ended video question answering. Given spatio-temporal network hsp (·), video v and question q, the spatio-temporal attention network learning with multi-step reasoning process is given by:

where is recursively updated. The joint representation of spatio-temporal attentional video is returned after the R-th update, given by f(q, v) = yR. The learning process of reasoning spatio-temporal attention networks is illustrated in Figure 2.

We now present the decoder neural network g(·) based on the joint representation of spatio-temporal attentional video for answer prediction. Unlike the encoder neural network which simply encodes the spatio-temporal attentional video representation according to the question, we decoder neural network is learned to generate the answer. At each time j, the decoder computes the probability of generating k-th word by

where w(o) ky and w(0) kh are parameter vectors. The h(0) j is the j-th latent state of the decoder neural network. We note that instead of the decoder layer to generate free-form answers for open-ended question answering, it is also possible to be extended for multiple-choice question answering.

3 Experiments

3.1 Data Preparation

We construct the video question-answering datset from the annotated video clip data [Li et al., 2016] with natural language descriptions, which consists of 201,068 GIFs and 287,933 descriptions. Following the state-of-the-art question generation method [Heilman and Smith, 2010], we generate the question-answer pairs from the movie descriptions. Following the existing question answering approaches [Antol et al., 2015; Shih et al., 2016; Yang et al., 2016; Zhao et al., 2015; 2016], we generate four types of questions, which are related to the object, number, color and location queries for the video. We split the generated dataset into three parts: the training, the validation and the testing sets. The four types of video question-answering pairs used for the experiments are summarized in Table 1.

We then preprocess the video question-answering dataset as follows. We first sample 25 frames from each video and then resize each frame to 224×224. We extract the visual representation of each frame by the pretrained VGGNet [Simonyan and Zisserman, 2014], and take the 4,096dimensional feature vector for each frame. We choose 3 candidate regions for each frame. We employ the pretrained word2vec model to extract the semantic representation of questions and answers. Specifically, the size of vocabulary set is 6,500 and the dimension of word vector is set to 256. Note that we add a token <eos> to mark the end of the answer phrase, and take the token <Unk> for the out-of-vocabulary word.

3.2 Evaluation Criteria

We evaluate the performance of our proposed r-STAN method based on two widely-used evaluation criteria for visual question answering, i.e., Accuracy [Antol et al., 2015] and WUPS [Malinowski and Fritz, 2014]. Given the testing question q ∈ Qt with its ground-truth answer a, we denote the predicted answers from our r-STAN method by o. We now introduce the evaluation criteria below.

• Accuracy. The Accuracy is the normalized criteria of accessing the quality of the generated answer based on the testing question set Qt, given by

where Accuracy = 1 (best) means that the generated answer and the ground-truth ones are exactly the same, while Accuracy = 0 means the opposite.

• WUPS. The WUPS is the soft measure based on the WUP [Wu and Palmer, 1994] score to evaluate the quality of the generated answer. The WUP measures word similarity based on WordNet [Fellbaum, 1998]. Thus, given the set of generated answer words Oq = {o1, o2, . . . , oM} and the ground-truth ones Aq = {a1, a2, . . . , aM} for testing question q, the WUPS score with the threshold γ is given by

where the WUPγ(·) score is given by

Following the experimental setting in [Malinowski and Fritz, 2014], we choose two WUPS evaluation criteria with the parameter γ to be 0 and 0.9, denoted by WUPS@0.0 and WUPS@0.9, respectively. Because of space limitation, we present the experimental results with M = 1, and illustrate the results with high value of M in the extended version of this paper.

3.3 Performance Comparisons

We extend the existing visual question answering methods as the baseline algorithms for the problem of video question answering.

• VQA+ method is the extension of VQA algorithm [Antol et al., 2015], where we add the mean-pooling layer that obtains the joint video representation from VGGNet-based frame features, and then computes the joint representation of question embedding and video representation by their element-wise multiplication for generating open-ended answers.
• SAN+ method is the incremental algorithm based on stacked attention networks [Yang et al., 2016], where we add the GRU network to fuse the sequential representation of spatially attended frames for video question answering.
• UTC+ method is based on the temporal-context encoder-decoder framework [Zhu et al., 2015], where we add the GRU-based decoder network for video question answering.
• QRU+ method is modified from the QRU algorithm [Li and Jia, 2016], where we add the bidirectional GRU network for obtaining video representation and perform multiple question representation updates for video question answering.

Among them, methods VQA+, SAN+ and QRU+ are extended from the image-based question answering methods, while UTC+ is modified from fill-in-the-blank video question answering method. Unlike the previous visual question answering works, our r-STAN method learns the hierarchical spatio-temporal attended video representation with multiple reasoning process for the problem. To exploit the effect of reasoning process, we denote the our r-STAN method with r reasoning steps by r-STAN(r) and the one without reasoning process by r-STAN(0). The input words of our method are initialized by pre-trained word embeddings [Mikolov et al., 2013] with size of 256, and weights of GRUs are randomly by a Gaussian distribution with zero mean.

Table 2 shows the overall experimental results of the methods on all types of questions based on three evaluation criteria. Tables 3, 4 and 5 illustrate the evaluation results on Accuracy, WUPS@0.0 and WUPS@0.9 with different types of questions, respectively. The hyperparamters and parameters which achieve the best performance on the validation set are chosen to conduct the testing evaluation. We report the average value of all the methods on three evaluation criteria. The experiments reveal a number of interesting points:

• The methods based on GRU network learning, SAN+, UTC+ and QRU+ outperform the mean-pooling based method VQA+, which suggests that the sequential frame-level representation is critical for the problem.
• The reasoning based method QRU+ achieves better performance than other baselines. This suggests that the reasoning framework that enables the multiple updates over the joint representation of video and question can also improve the performance of video question answering.
• In all the cases, our r-STAN method achieves the best performance. This fact shows that the reasoning spatiotemporal attention network learning framework that exploits both the joint spatio-temporally attended video representation, and multiple reasoning updates can further improve the performance of video question answering.

In our approach, there are three essential parameters, which are the dimension of hidden state in bi-GRU networks, the dimension of hidden state in bi-aGRU network and the size of fully connected units for decoder networks. We investigate the effect of these parameters on our method by varying both the dimension of hidden state in bi-GRU and bi-aGRU networks from 128 to 1,024, and the number of hidden units in fully connected layer from 300 to 1,200. We first illustrate the performance of our method by varying the dimension of hidden state in bi-aGRU network, the dimension of hidden state in bi-GRU network and the number of hidden untis on Accuracy in Figures 3(a), 3(b) and 3(c). We then vary these parameters to show their effect on our method using WPUS@0.9 in Figures 4(a), 4(b) and 4(c). Our method achieves the best performance when the dimension of hidden state of bi-GRU networks is set to 512, the dimension of hidden state in biaGRU networks is set to 512 and the number of hidden units in fully connected layer is set to 500.

4 RelatedWork

In this section, we briefly review some related work on imagebased question answering and video-related question answering.

The problem of image-based question answering has attracted considerable attention recently [Antol et al., 2015; Shih et al., 2016; Lu et al., 2016; Li and Jia, 2016; Xiong et al., 2016]. Given an image and a natural language question about the image, the task of image question answering [Antol et al., 2015] is to provide an accurate natural language answer. With the advancement of visual attention [Xu et al., 2015], Shih et. al. [Shih et al., 2016] introduce the spatial attention that selects the relevant image regions to the textbased questions. Yang et. al. [Yang et al., 2016] develop the stacked attention networks and Lu et. al. [Lu et al., 2016] propose the co-attention mechanism with joint image and question attention for image question answering. To exploit complex visual question answering tasks, QRU method [Li and Jia, 2016] employs the reasoning process with attention mechanism that iteratively selects the relevant image regions for question representation update. Xiong et. al. [Xiong et al., 2016] introduce the dynamic memory networks for both image and textual question answering. Kim et. al. [Kim et al., 2016] present multimodal residual networks for image question-answering, which uses element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models. Malinowski et. al. [Malinowski and Fritz, 2014] propose a multi-world approach for openended image question answering. A survey of existing image question answering works can be found in [Wu et al., 2016].

As a natural extension of image-based question answering, video-related question answering is introduced as a more challenging task, which has drawn a significant attention [Tapaswi et al., 2016; Zhu et al., 2015; Mazaheri et al., 2016; Tu et al., 2014; Yu et al., 2015]. Tapaswi et. al. [Tapaswi et al., 2016] introduce the multi-modal movierelated question answering with high-level abstract concepts, which are extracted from both visual and external information. On the other hand, the proposed models [Zhu et al., 2015; Mazaheri et al., 2016; Tu et al., 2014] for fill-inthe-blank [Yu et al., 2015] video-related question answering task are mainly based on time-invariant visual information, which are extended from the existing image-based question answering methods. Unlike the previous studies, we formulate the problem of open-ended video question answering from the viewpoint of hierarchical spatio-temporal attentional encoder-decoder network learning.

5 Conclusion

In this paper, we present the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We first propose the hierarchical spatio-temporal attention networks to learn the video representation from the critical frames of the targeted objects according to the question. We then incorporate the multi-step reasoning process to our proposed attention networks that enables the progressive joint representation learning of multimodal spatio-temporal attentional video and textual question for video question answering. We construct a large-scale video question answering dataset and evaluate the effectiveness of our proposed method through extensive experiments.

Acknowledgments

This work was supported by National Basic Research Program of China (973 Program) under Grant 2013CB336500, and National Natural Science Foundation of China under Grant 61602405, Fundamental Research Funds for the Central Universities 2016QNA5015 and the China Knowledge Centre for Engineering Sciences and Technology. The Project is also Supported by the Key Laboratory of Advanced Information Science and Network Technology of Beijing (XDXX1603).