Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks

1 Introduction

Video question answering is the visual information delivery mechanism that enables user to issue their queries and then collect the answers from the referenced visual contents. Open-ended video question answering is the essential problem of visual question answering, which automatically generates the natural language answer from the referenced video contents according to the given question. Currently, most of the video question answering approaches mainly focus on the problem of short-form video question answering [Zeng et al., 2017; Zhao et al., 2017; Jang et al., 2017], which learn the semantic video representation from LSTM network layer, and then generate the answer to the given question. Although the existing works have achieved promising performance in short-form video question answering, they may still be ineffectively applied to the long-form video question answering due to the lack of modeling the semantic representation of long-form video contents.

The long-form video contents often contain the evolving complex object interactions through frames, which have longterm semantic dependencies [Lezama et al., 2011]. We illustrate a simple example in Figure 1. We show that the answer generation to question "what is the woman holding?" requires the semantic localization of the targeted segment from the long-form video contents. Thus, the simple extension of the existing video question answering works based on framelevel LSTM networks is difficult for modeling the semantic representation of long-form video contents according to the given question [Krishnan and Sitaraman, 2013]. Recently, hierarchical neural encoder [Pan et al., 2016] has been proposed to learn the segment-level video semantic representation with fixed segment length. We then employ the hierarchical neural encoder with attention mechanisms to learn the joint representation of video contents according to the given question. On the other hand, although the video frames are topically consistent, they have different semantic contents [Lezama et al., 2011]. Inspired by binary neurons [Bengio et al., 2013], we then develop the hierarchical neural encoder that unifies the adaptive video segmentation and joint representation modeling of video content according to the given question into a joint learning framework. Thus, leverage the hierarchical neural encoder with adaptive video segmentation is critical for modeling the semantic representation of long-form video contents for video question answering.

In this paper, we study the problem of open-ended longform video question answering from the viewpoint of adaptive hierarchical reinforced encoder-decoder network learning. We propose the hierarchical neural encoder with adaptive recurrent network to learn segment-level question-aware video representation with adaptive video segmentation. We devise the reinforced decoder network to generate the natural language answer for open-ended video question answering. We then develop the adaptive hierarchical reinforced network learning framework, named as AHN. When a certain question is issued, AHN can generate natural language answer for it based on the referenced video contents. The main contribution of this paper are as follows:

• Unlike the previous studies, we study the problem of open-ended long-form video question answering from the viewpoint of adaptive hierarchical reinforced encoder-decoder network learning.
• We develop the adaptive hierarchical encoder to learn the segment-level question-aware video representation with adaptive video segmentation. We then devise the reinforced decoder to generate the answer for openended video question answering.
• We construct a large-scale dataset for open-ended longform video question answering and validate the effectiveness of our propose method through extensive experiments.

2 Video Question Answering via Adaptive Hierarchical Reinforced Networks

2.1 The Problem

Before presenting the learning framework, we first introduce some basic notions and terminologies. We denote the question by q ∈ Q, the video v ∈ V and the answer by a ∈ A, where Q, V and A are the sets of questions, videos and answers, respectively. Since the video is composed of sequential frames, the frame-level representation of video v is given by v = (v1, v2, . . . , vN) of length N, where v2 is the second frame. We then encode the word-level representation of natural language answer by a = (a1, a2, . . . , aM) of length M, where aM is the M-th word token. We then denote the collection of video segments by {S1, S2, . . . , SK} of size K, where S2 is the set of segmented frames and vi ∈ S2 means that the i-th frame belongs to segment S2. The semantic representation of segment Sk is denoted by sk, and the sequential representation of video segments is given by (s1, s2, . . . , sK). Since both the video and answer are sequential data with variant length, it it natural to choose the variant recurrent neural network called long-short termmemory network (LSTM) [Hochreiter and Schmidhuber, 1997] to learn their feature representations by

where δ represents the sigmoid activation function; Ws, Gs and Vo are the weight matrices, and bs are the bias vectors. The memory cell ct maintains the history of the inputs observed up to the timestep. Update operations on the memory cell are modulated by three gates it, ft and ot, which are all computed as a combination of the current input xt and of the previous hidden state ht-1, followed by a sigmoid activation. Specifically, we denote the semantic representation of video v by h(v) = (h(v) 1 , h(v) 2 , . . . , h(v) N ) and that of answer a by h(a) = (h(a) 1 , h(a) 2 , . . . , h(a) M ) using LSTM networks.

Using the notations above, the problem of open-ended video question answering is formulated as follows. Given the set of videos V , questions Q and answers A, our goal is to learn the encoder-decoder network model g(f(v, q)) where the encoder network f(v, q) that learns the joint representation of the video and question, and the decoder network ˆa = g(f(v, q)) generates the answer ˆa for open-ended video question answering.

2.2 Adaptive Encoder Network Learning

In this section, we propose the adaptive encoder network f(v, q) that unifies the hierarchical video segmentation and semantic representation learning into a common framework.

The video contents often contain a number of frames with the targeted objects to the given question that evolves over time [Yao et al., 2015]. Inspired by binary neuron [Bengio et al., 2013], we propose the adaptive attentional recurrent neural networks that segment the complex events in video to capture their long-term semantic dependencies, and learn the joint representation of relevant frames and segments according to the question. We first extract the frame-level video feature using ConvNet [Simonyan and Zisserman, 2014] by v(f) = (v(f) 1 , v(f) 2 , . . . , v(f) N ), and then learn their semantic representation using LSTM networks. When the end frame of a video event is estimated, we reset the LSTM parameters of the next frame in order to segment the video. To enable the video segmentation, we define an adaptive recurrent neural networks with binary gate function, which decides whether to transfer the LSTM parameters (i.e., hidden state h(v) t and memory cell c(v) t ) of the current frame to update the LSTM parameters of the next frame (i.e., hidden state h(v) t+1 and memory cell c(v) t+1 ) by Equations (1), (2), (3), (4) and (5) or reinitialize them. Formally, the t-th binary gate is defined as a step function, which is computed as a non-linear combination of the feature of the t+1-th frame from ConvNet (i.e., v(f ) t+1 ), and the semantic representation of the t-th frame from LSTM networks (i.e., h(v) t ), given by

The 1[·] is a step function and δ(·) is a sigmoid function. The wγ is a learnable row vector, Wγv, Wγh and bγ are the learnable weights and bias. The τ is the threshold parameter of the step function. For example, the inputs of the 3-th binary gate γ3 are the h(v) 3 (i.e., the hidden state of the 3-th frame from LSTM networks) and v(f) 4 (i.e., the video feature of the 4-th frame) in Figure 2. That is, the binary gate function γt decides whether to transfer the parameters of the t-th frame from LSTM networks by h(v) t ← h(v) t · (1 - γt) and c(v) t ← c(v) t · (1-γt), respectively.

Given the semantic representation of frames (h(v) 1 , h(v) 2

,. . ., h(v) N ) with binary gate values (γ1, γ2, . . ., γN-1), we then learn the joint question-aware video segment representation. If the value of binary gate γt = 1, the question-aware representation of current segment Sk is computed and then passed to the segment-level LSTM networks. Given the question representation h(q) , the frame-level attention for the t-th frame vt ∈ Sk is given by

where W(f) h , W(f) q are parameter matrices and b(v) is the bias vector. The P(f) is the parameter vector for computing the frame-level attention score. For each frame vt ∈ Sk, its activation by the softmax function is given by β(v) t = exp(α(v) t ) ∑vt∈Sk α(v) t . Thus, the attentional semantic representation

for segment Sk is then given by sk = ∑vt∈Sk β(v) t h(v) t . Given the representation of segments (s1, s2, . . . , sK), we devise the segment-level LSTM networks to learn their semantic representation, denoted by (h(s) 1 , h(s) 2 , . . . , h(s) K ). Therefore, the adaptive encoder network is given by f(v, q) = (h(s) 1 , h(s) 2 , . . . , h(s) K ).

2.3 Reinforced Decoder Network Learning

In this section, we propose the reinforced decoder network g(·) based on segment-level LSTM networks to generate the open-ended answer for video question answering.

Given the segment-level semantic representation of video contents f (v, q) = (h(s) 1 , h(s) 2 , . . . , h(s) K ) from segment-level LSTM networks, the question-aware attentional LSTM networks predict the next word in answer by sampling at ∼ pθ(at|a1:t-1, f (v, q)) = gt(f (v, q), h(a) t , et), where gt(·) is the recurrent answer generator at time t. The h(a) t is the decoder state and et is the context vector at time t. Given question representation h(q) , the segment-level attention score for the k-th segment and t-th decoder state for answer generation is given by

where W(s) h , W(s) q , W(a) h are parameter matrices and b(s) is the bias vector. The P(s) is the parameter vector for computing the question-aware segment-level attention score. For each segment h(s) k with decoder state h(a) t , its activation by the softmax function is given by β(s) k = exp(α(s) k ) ∑k exp(α(s) k ) . Thus, the attentional question-aware semantic video representation at decoder state h(a) t is given by et = ∑k β(s) k h(s) k . One common approach to train the proposed decoder network is under the framework of maximum likelihood estimation, given by

However, the training based on maximum likelihood estimation makes the learnt decoder network suboptimal [Bahdanau et al., 2016]. In this work, we train the proposed decoder network under the framework of reinforcement learning. In the setting of reinforcement learning, we define the generation of next answer word as action, and the decoding probability pθ(at|a1:t-1, f(v, q)) as the policy. Following the existing visual-semantic embedding works [Ren et al., 2017], we choose the reward function based on the embedding similarity between the ground-truth answer a and the generated answer ˆa, given by Ra(ˆa) = ‖h(a) - h(ˆa) ‖2 . Specifically, we define the expected cumulative reward at each decoding step using value function by Q(ˆat|ˆa1:t-1, f(v, q)) = Epθ(ˆat+1:M|ˆa1:t,f(v,q))Ra(ˆa). The value function Q(ˆat|ˆa1:t-1, f(v, q)) is then estimated by aggregating the Monte-Carlo simulation at each decoding step, given by

The {ˆa(1) t+1:M , ˆa(2) t+1:M , . . ., ˆa(J) t+1:M } is the set of generated answers, which are randomly sampled starting from the t+1-th decoding step using current state and action. The gradients of the reinforced decoder network according to the policy gradient theorem is given by

3 Experiments

3.1 Data Preparation

We construct the long-form video question answering dataset from the ActivityNet data [Heilbron et al., 2015] with natural language descriptions, which contains 20,000 videos amounting to 849 hours and 100,000 descriptions. The average time duration of videos is around 180 seconds and the longest video runs for over 10 minutes. Following the state-of-the-art question generation method [Heilman and Smith, 2010], we generate the question-answer pairs from the video descriptions. Following the existing visual question answering approaches [Antol et al., 2015; Shih et al., 2016; Zhao et al., 2017], we generate five types of questions, which are related to the object, number, color, location and action for the video. We split the generated dataset into three parts: the training, the validation and the testing sets. The five types of long-form video question-answering pairs used for the experiments are summarized in Table 1. The dataset will be provided later.

We then preprocess the long-form video question answering dataset as follows. We first resize each frame to 224×224 and then extract the visual feature of each frame by pretrained VGGNet [Simonyan and Zisserman, 2014], and take the 4096-dimensional feature vector for each frame. We employ the pretrained word2vec model to extract the semantic representation of questions and answers. Specifically, the size of vocabulary set is 8,500 and the dimension of word vector is set to 256. Note that we add a token <eos> to mark the end of the answer, and take the token <unk> for the out-ofvocabulary word.

3.2 Performance Criteria

We evaluate the performance of our proposed AHN method based on two widely-used evaluation criteria for open-ended visual question answering, i.e., Accuracy [Antol et al., 2015] and WUPS [Malinowski and Fritz, 2014]. Given the testing question q ∈ Qt with its ground-truth answer a, we denote the generated answers from our AHN method by o. We now introduce the evaluation criteria below.

• Accuracy. The Accuracy is the normalized criteria of accessing the quality of the generated answer based on the testing question set Qt, given by

where Accuracy = 1 (best) means that the generated answer and the ground-truth ones are exactly the same, while Accuracy = 0 means the opposite.

• WUPS. The WUPS is the soft measure based on the WUP [Wu and Palmer, 1994] score to evaluate the quality of the generated answer. The WUP measures word similarity based on WordNet [Fellbaum, 1998]. Thus, given the set of generated answer words Oq = {o1, o2, . . . , oM} and the ground-truth ones Aq = {a1, a2, . . . , aM} for testing question q, the WUPS score with the threshold γ is given by

where the WUPγ(·) score is given by

Following the experimental setting in [Malinowski and Fritz, 2014], we choose two WUPS evaluation criteria with the parameter γ to be 0 and 0.9, denoted by WUPS@0.0 andWUPS@0.9, respectively.

3.3 Performance Comparisons

We compare our proposed method with other four state-ofthe-art methods for the problem of open-ended video question answering as follows:

• VQA+ method is the extension of image question algorithm [Malinowski and Fritz, 2014], where the meanpooling layer is added to encode the video.
• MM+ method [Zeng et al., 2017] is the extension of end-to-end memory network algorithm, which one-layer bi-LSTM network is added to encode the sequence of video frames for answer generation.
• STAN method [Zhao et al., 2017] is based on the hierarchical spatio-temporal attention network for learning the joint representation of the dynamic video contents according to the given question.
• STVQA method [Jang et al., 2017] is based on the spatio-temporal reasoning algorithm, which employ the spatial and temporal attention on video to answer questions.

Unlike the previous video question answering works, our AHN method learns the hierarchical attentional video representation with adaptive recurrent encoder networks, and then generates the natural language answer with reinforced decoder networks for open-ended video question answering. To exploit the effect of reinforced decoder networks, we denote the our AHN method with reinforced decoder networks by AHN(rl) and the one without reinforced decoder networks by AHN(ml). The weights of both frame-level LSTM networks and segment-level LSTM networks are randomly by a Gaussian distribution with zero mean.

Table 2 shows the overall experimental results of the methods on all types of questions based on three evaluation criteria. Tables 3, 4 and 5 illustrate the evaluation results on Accuracy, WUPS@0.0 and WUPS@0.9 with different types of questions, respectively. The hyperparamters and parameters which achieve the best performance on the validation set are chosen to conduct the testing evaluation. We report the average value of all the methods on three evaluation criteria. The experiments reveal a number of interesting points:

• The methods based on LSTM networks, MM+, STAN, STVQA, AHN outperform the mean-pooling based method VQA+, which suggests that the sequential frame-level representation is critical for the problem.
• The attention based methods STAN, STVQA and AHN achieves better performance than other baselines. This suggests that the joint representation learning of video and question can also improve the performance of openended video question answering.
• In all the cases, our AHN method achieves the best performance. This fact shows that the hierarchical attentional video representation with adaptive recurrent encoder networks, and reinforced decoder networks are effective for the problem.

In our approach, there are three essential parameters, which are the dimension of hidden state in frame-level LSTM networks, the dimension of hidden state in segment-level LSTM networks and the threshold τ of the step function . We investigate the effect of these parameters on our method by varying both the dimension of hidden state in frame-level LSTM networks and segment-level LSTM networks from 128 to 1024, and the threshold τ of step function γt(v(f) t+1 , h(v) t ) from 0.1 to 0.9 on Accuracy in Figures 3(a), 3(b) and 3(c). We then vary these parameters to show their effect on our method using WPUS@0.9 in Figures 4(a), 4(b) and 4(c). Our method achieves the best performance when the dimension of hidden state in frame-level LSTM networks is set to 512, the dimension of hidden state in segment-level LSTM networks is set to 256 and the threshold is set to 0.3.

4 RelatedWork

In this section, we briefly review some related work on visual question answering.

The existing approaches for visual question answering can be categorized into image-based question answering methods [Antol et al., 2015; Lu et al., 2016; Li and Jia, 2016; Malinowski and Fritz, 2014; Shih et al., 2016] and videobased question answering ones [Mazaheri et al., 2016; Tapaswi et al., 2016; Zhu et al., 2015; Zeng et al., 2016; Zhao et al., 2017]. Given a natural-language question for the image, the task of image-based question answering is to provide the accurate answer for the given question [Antol et al.,

2015]. Malinowski et. al. [Malinowski and Fritz, 2014] develop the multi-world probabilistic approach for open-ended image question answering. With the development of attention mechanism, Shih et. al. [Shih et al., 2016] propose the spatial-attention mechanism that selects the relevant image regions to the given question. Lu et. al. [Lu et al., 2016] devise the co-attention mechanism and Yang et. al. [Yang et al., 2016] develop the stacked attention method for image question answering. To exploit the complex image question answering task, QRU method [Li and Jia, 2016] is proposed with reasoning process that iteratively selects the relevant image regions and updates the question representation. A survey of existing image question answering methods can be found in [Wu et al., 2016].

As a natural extension of image-based question answering, the video-based question answering has been proposed as a more challenging task [Zeng et al., 2016]. The fill-in-theblank approaches [Zhu et al., 2015; Mazaheri et al., 2016] complete the missing entry in the video description by ranking candidate answers based on both visual content and contextual video description. Tapaswi et. al. [Tapaswi et al., 2016] propose the three-way scoring function for movie question answering based on both the relevance between given question and textual movie subtitles, and textual movie subtitles and answers. Zhao et. al. [Zhao et al., 2017] propose the hierarchical spatio-temporal attention networks for video question answering. Jang et al. [Jang et al., 2017] develop the spatio-temporal reasoning algorithm, which employ the spatial and temporal attention on video to answer questions. Zeng et al. [Zeng et al., 2017] extend the end-to-end memory network with additional LSTM layer for video question answering. Although these works have achieved promising performance in short-form video question answering, they may still be ineffectively applied to the long-form video question answering due to the lack of modeling the semantic representation of long-form video contents. Unlike the previous studies, we study the problem of open-ended long-form video question answering from the viewpoint of adaptive hierarchical network learning.

5 Conclusion

In this paper, we present the problem of open-ended longform video question answering from the viewpoint of adaptive hierarchical reinforced encoder-decoder network learning. We first propose the adaptive hierarchical encoder network to learn the video representation from the critical segments of the targeted objects with adaptive video segmentation. We then develop the reinforced decoder network to generate the natural language answer for open-ended video question answering. We construct a large-scale long-form video question answering dataset and evaluate the effectiveness of our proposed method through extensive experiments.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant No.61602405, No.61702143, No.61622205 and No.61472110, Sponsored by CCF-Tencent Open Research Fund and the China Knowledge Centre for Engineering Sciences and Technology. This work is also Supported by Zhejiang Natural Science Foundation(LZ17F020001) and Key R&D Program of Zhejiang Province(2015C01027).