A Question Type Driven Framework to Diversify Visual Question Generation

1 Introduction

Recent years see the popularity of multi-modal research on vision and language. Popular tasks include visual caption generation (VCG) [Vinyals et al., 2015] and visual question answering (VQA) [Antol et al., 2015]. VCG aims at generating descriptions for a given image with a goal of scene understanding, while VQA provides a related question and requires an answer to it. Research for these two tasks are fueled by several manually generated corpora [Lin et al., 2014; Zhu et al., 2016]. Different from generating a statement (descriptions or answers), visual question generation (VQG) aims at asking questions about the given image. Teaching machine the skill of asking is important in a variety of areas, e.g., providing demonstrations in child education [Kunichika et al., 2004], initializing a conversation for chatbots [Mostafazadeh et al., 2017], etc. On the other hand, it can benefit the question answering task by constructing question sets automatically [Ren et al., 2015] to reduce the labor of human annotation.

∗Corresponding author

VQG is a rising research topic in both fields of computer vision and natural language processing [Ren et al., 2015; Mostafazadeh et al., 2016]. Mostafazadeh et al. [2016] explored different approaches for this task, and the experimental results showed a retrieval based approach which chose the question from the closest image achieved the best performance. In their experiment setting, only one question is generated for a given image. We argue that different people might have different questions about the same image, therefore, a visual question generation system should also be able to produce questions with various of focuses. Although there are some attempts to diversify the results for text generation [Vijayakumar et al., 2018; Jain et al., 2017], none of them considers the special characteristics of questions.

Question is a linguistic expression used to make requests for information. In terms of information needs, we can classify questions into different types, such as, what, which, how, etc. Figure 1 presents two sample images and each of them is accompanied with five human generated questions. We have two observations: 1) Questions related to a given image have various types. 2) There can be several questions for each single type with different focuses. Further analysis on a human generated image-question paired dataset (VQG-Flickr) from [Mostafazadeh et al., 2016] shows that around 52.8% of images are questioned by more than 2 types of questions. It is deemed that question type can be used to organize questions for different information needs. Thus we explore the problem that how question type can be used to enable diverse question generation.

In this paper, we propose a question type guided strategy for diverse question generation. In our framework, a question is constructed in two steps. First, a question type is sampled to determine what kind of information is requested. Second, the content of the question is generated conditioning on the sampled question type and the visual information of the image. Two components are proposed to enforce the question diversification.

• Variational auto-encoder based question generator. For each question type, multiple questions can be asked with different focuses. Instead of using a deterministic sequence-to-sequence model [Cho et al., 2014], we propose a conditional variational auto-encoder (C-VAE) to produce multiple questions for a specific question type.
• Question type distribution learner. The probability of different question types should be different according to the content of images. For example, questions with the type who might be less possible for a landscape picture. To utilize this intrinsic characteristic, we design a model

to estimate the probability distribution of the question types for the input image.

A neural network based framework is proposed to conduct the learning of these two components jointly. Questions are selected by considering both the generation probability from C-VAE and the probability of the corresponding question type. We evaluate our framework on three public benchmark datasets in terms of relevance and diversity. Experimental results show that our framework outperforms the state-ofthe-art visual text generation models in a large margin.

2 Model

Our model takes an image Ii as input and generates diverse questions as output. The overall framework is shown in Figure 2. It consists of four components, namely question type and image encoder, question type distribution learner, question generator, and question selector. The question type and image encoder learns the mixed representations for question types and the given image as the input for the other components. The question type distribution learner computes the probability distribution of different question types. The question generator produces multiple questions for a specific question type based on C-VAE. Finally, the question selector outputs top-k questions considering both the generation probability from C-VAE and the probability of corresponding question type.

2.1 Question Type and Image Encoder

Interrogative words (e.g. who, which, etc.) in questions imply the type of the information that questioners want to acquire.

It is also widely used as a way to group questions [Zhu et al., 2016]. Although there are many alternative taxonomies to organize questions [Graesser et al., 2008], they also involve heavy human annotation. For simplicity, we directly use interrogative words to represent question types in our framework. Note that the model is compatible for other question classification schemes.

Based on interrogative words, we predefined N question types. Each question type is represented as an embedding vector with fixed length. Embedding vectors are initialized by the word embeddings (from word2vec) of their corresponding interrogative words and would be optimized in the training process. We use VGGNet [Simonyan and Zisserman, 2015] to process the image and borrow fc7 1 feature to represent the image. In order to learn the correlation between the question type and the image feature, we concatenate them and feed them into a three-layer MLP (multilayer perceptron) with batch normalization. The procedure can be described by ci,j = f1([vi, tj ]). Inwhich, f1 stands for the corresponding MLP transformation, [·] stands for the concatenating operation and ci,j is the processed feature vector related to image i and the question type tj.

2.2 Question Generation via Conditional Variational Auto-Encoder

Our question generator aims to produce questions for the given image Ii with a specific question type tj. Given the mixed representation cij for Ii and tj, a recurrent neural network (RNN) is used to decode the source information into a sequence of words to form a question. Considering that there can be multiple questions about an image with a specific question type, a deterministic decoder is not satisfactory for diverse question generation.

Recently, variational auto-encoders (VAEs) [Kingma and Welling, 2014; Rezende et al., 2014] showed strong capability in modeling latent random variables and improved the performance for generation tasks on both text [Bowman et al., 2016; Li et al., 2017] and image [Gregor et al., 2015]. Inspired by [Jain et al., 2017], we employed a conditional variational auto-encoder (C-VAE) to generate multiple questions for a specific question type. In our case, questions are generated in condition on the given image and a question type. A specific question type can be interpreted as a pre-defined cluster for the VAE to generate questions from a fine-grained setting which can further enforce diversity. In other words, this modification provide a guidance for VAE generators.

A C-VAE consists of two components: variational encoder (inference) and variational decoder (generation). During the inference, we map the ground-truth questions into a latent space and learn to recover it in the generation process. A distribution representing the latent space is optimized in the training process, from which we sample a latent variable z for generation. Since the latent variable follows some distribution instead of being a deterministic value, the decoder is able to generate different questions via sampling. In inference and generation processes, visual and the question type information (denoted by cij) are used as condition to guide latent distribution learning and question generation respectively. The framework of our C-VAE is shown in Figure 3.

1 fc7 is the output of 7th fully connected layer of VGGNet.

Suppose that the target latent distribution for the image Ii and the question type tj is Pθ(zi,j|Ii, tj). We aim to learn a distribution Pφ(zi,j|Ii, tj) to approximate the target distribution Pθ(zi,j |Ii, tj). We can have the following equations:

Given the latent variable follows a normal distribution N(0, I) in VAE, the first term of the last expression in Equation 1 can be marginalized. And the second term can be estimated by drawing samples from the distribution Pφ(zi,j|qi,j, Ii, tj). The lower bound of log(Pθ(qi,j|Ii, tj)), which is also the loss of variational auto-encoder, can thus be written as Equation 2:

where z(l) i,j = gφ(qi,j, Ii, tj, ϵi,j), ϵi,j ∼ N(0, I), and L is the number of samples drawn.

We make use of a RNN gφ to draw samples from the distribution Pφ(zi,j|qi,j, Ii, tj), from which we can get a final state hTi,j ∈ Rdf . Two fully connected layer f2 and f3 are then used to transfer hTi,j into ui,j and logσ2 i,j which are the mean and logarithm variance of some Gaussian distribution. The process can be represented by Equation 3.

Different from the traditional variational auto-encoder that only uses the variables sampled from the latent space for decoding, we need to take the visual information into consideration. In the generation component, feature vector ci,j (information for the question type tj and the image i) and the sampled variable zi,j are fed into the first state and second state of the RNN decoder respectively. In the training stage, zi,j is derived from function gφ, through which the model encodes generated questions into the latent space and approximates the prior N(0, I). In the testing stage, zi,j is directly sampled from the learned latent space. In order to get more diverse questions, we can sample from a probability distribution that has larger variance such as N(0, 5I). Then the sampled value is regarded as the latent variable zi,j.

2.3 Question Type Distribution Learner

Given an image, question type distribution learner produces a probability distribution to indicate how likely the image will be inquired by different question types. As this involves all question types, we stack all the ci,j across different question types together to form a matrix with a dimension of N × df . The obtained matrix is then fed into the three-layer MLP transformation f1 (shared with the one used in the representation component) and a three-layer convolution component in sequence. Two fully connected layers followed by a softmax layer are then used to estimate the final question type distribution for the given image.

A cross-entropy based loss function is utilized for parameter learning. The loss for image Ii in terms of question type probability prediction is shown as Equation 4:

where pi,j and ˆ pi,j denote the target and predicted probability for question type tj respectively. We use the maximum likelihood estimation to compute the target probability of question type tj for image Ii.

Note that both question type distribution learner and question generator are optimized jointly with the objective function:

where λ is introduced to balance the contribution of these two loss items.

2.4 Question Selection

For the given image, we assign a probability to each generated question and select top questions as the output. Taking question type tj into consideration, the probability of question qi,j generated for image Ii can be expressed as Equation 6:

where P(tj|Ii) stands for the probability of generating questions of type tj and it can be estimated by the result from Section 2.3 directly. P(qi,j|Ii, tj) stands for the generation probability of the constructed question and it can be computed in the decoding process of the C-VAE.

In the process of question generation, our target is the following expression:argmax

where qi,j is decodedword byword based on a RNNdecoder. Suppose wi,j,t is the word in the tth time step for for generating qi,j , the target for each time step in decoder can be viewed as the following expression:

And the likelihood of the generated question can be calculated by multiplying all the probability elements of each word:

Usually, log-likelihood is used in practice. We can compute every single probability P on the right side through every time step in decoder. Obviously, maximizing every single probability cannot guarantee maximizing the final target. Therefore, we use beam search [Wu et al., 2017] to obtain max{P (qi,j |Ii, tj )}.

We use S1 to represent the probability of generating target question based on p(qi,j |Ii, tj ):

In order to avoid the problem that such model favors shorter question, a length penalty lp(qi,j) is introduced and it can be computed as Equation 11 following [Wu et al., 2017]:

where |qi,j| represents the number of words in the generated question.

In line with S1, we simply take logarithmic on the probability of the corresponding question type given the image which is represented as S2:

During the testing, we consider scores from question generation (S1) and question type probability (S2) to compute the final score of generated questions. Based on such score, generated questions are able to be ranked, and we can select top k from them as output.

3 Experiment

3.1 Experiment Datasets

We use three public datasets to evaluate our models, namely VQG-MS COCO, VQG-Flickr [Mostafazadeh et al., 2016] and Visual7W-telling [Zhu et al., 2016]. First two datasets contain 5,000 images with 5 human generated questions for each image and the third one has 28,653 images with various number of questions (refer to [Zhu et al., 2016] for details). In Visual7W-telling, questions are classified into 6 types based on interrogative words ('what', 'who', 'where', 'when', 'why', 'how'). With the same strategy, we categorize questions in VQG-MS COCO and VQG-Flickr into different types. By merging some interrogative words with different tenses into the same group (e.g., treat 'was' and 'is' as 'is'), we obtain 9 types ('what', 'who', 'where', 'when', 'why', 'how', 'is', 'do', 'can').

3.2 EvaluationMethods

Our model aims to generate multiple questions for a given image without sacrificing the quality of each single question. Therefore, we conduct two kinds of metrics to evaluate the performance of our model in terms of relevance and diversity respectively. For relevance evaluation, we report BLEU-4 [Papineni et al., 2002], corpus-BLEU-4, METEOR [Denkowski and Lavie, 2014] and ROUGE [Lin, 2004]. For diversity evaluation, we utilize mBLEU [Wang et al., 2016]. It assumes that the system is better if similarities among questions generated for an image are lower. Suppose that B(h,R) denotes the function of BLEU, and Qi = {qi,k, k = 1, 2, · · · ,K} is the K questions generated for image Ii, mBLEU can be computed by the Equation 14.

3.3 Comparative Models

We compare our models with some baselines and some stateof-the-art methods.

• NN-generator (I): Use the question from the most similar image as the question for a target image [Mostafazadeh et al., 2016]. Cosine similarity based on fc7 features is used to search for similar images. Note that only one question is generated by this model following its original setup.
• i2q (II): This is the state-of-the-art approach for text generation that generates a question from image features based on a deterministic sequence-to-sequence model [Ren et al., 2015]. Only one question is generated for each image.
• i2q+C-VAE (III): This is the model proposed by Jain et al. [2017] that uses C-VAE for questions generation.
• i2q+QT (IV): In addition to i2q, we introduce question type to guide question generation. Only generation probability S1 (refer to equation 10) is used to select top questions. This model is similar to the one proposed in [Shijie et al., 2017]. Note that only one question is generated with a specific type of question.
• i2q+QT+QTD (V): On top of i2q+QT, question type probability distribution is learned to guide question selection.
• i2q+QT+C-VAE+QTD (VI): This is the complete version of our model that uses both C-VAE for multiple question generation and selects top questions with the guidance of question type probability distribution.

3.4 Results and Analysis

Since some of the comparative models are unable to generate multiple questions, we use the top-1 question from multiple question generators for comparison. The overall results in terms of relevance can be seen in Table 1. We have several findings:

• The performance of NN-generator that uses information retrieval based approach is quite competitive in-line with the results reported in [Mostafazadeh et al., 2016]. However, further analysis on questions generated by NN-generator reveals that a large percentage of generated questions are not relevant to the target image. Therefore, the strategy of reusing questions from similar images is not sufficient for generating question with high quality.
• The performance of i2q is better than i2q+QT. Without considering question type, i2q is capable to compare questions across question types to select the one with highest probability. However, the question generated by i2q+QT is guided by question type that makes the probability comparison of questions across question types indirectly.
• By adding the probability distribution of question types to guide the top question selection, both i2q+QT+QTD and i2q+QT+C-VAE+QTD outperform i2q. This proves that our question type distribution learner is able to learn

the intrinsic characteristic of the given image. Besides, it can evaluate the appropriateness of questions from different question types in a global way to select questions with higher relevance.

• Our proposed model i2q+QT+C-VAE+QTD that uses both components of question type distribution learner and CVAE based question generator achieves the best performance in terms of all the four relevance metrics across three datasets. This confirms the effectiveness of our framework in terms of relevance.

We further evaluate the performance of our models in terms of diversity. We vary the number of questions generated by three multiple question generators to see how good they are for diverse question generation in terms of mBLEU (the less the better). Experimental results are shown in Figure 4. As the number of questions generated increases, mBLUE increases. By adding question type distribution learner, i2q+QT+QTD can generate more diverse questions than i2q+QT. With the help of C-VAE, i2q+QT+CVAE+QTD can improve the diversity further.

An example image with questions generated by different models is shows in Figure 5. Although i2q+QT is able to generate questions with different question types, the focus of questions are largely concentrated on the word temple. This is because the model would receive higher score for generating such a key term. With the guidance of question type distribution learner, questions generated by i2q+QT+QTD are more diverse in terms of topics. By adding C-VAE, i2q+C- VAE+QT+QTD is able to generate two questions for the type of what and improve the quality of the questions further. It is also exciting that our model can generate some good questions that are not mentioned in the ground-truth.

4 Conclusion and FutureWork

In this paper, we propose a neural-network based framework for visual question generation using interrogative words as question type to organize questions and enforce the diversification of the results. Experimental results on three publicly available question generation datasets showed the effectiveness of our framework in terms of both relevance and diversity. Further analysis on datasets shows that questions can be generated by considering information from multiple zones in the input image. Therefore, a feature extractor that treats all information equally is not able to manipulate input a such a fine-grained way. We thus will explore to use an attention mechanism to select and combine salient zones of a given image for question generation in future.

Acknowledgments

The work is partially supported by National Natural Science Foundation of China (Grant No. 61702106), Shanghai Science and Technology Commission (Grant No. 17JC1420200, Grant No.17YF1427600 and Grant No. 16JC1420401).