Generative Visual Dialogue System via Weighted Likelihood Estimation

1 Introduction

Artificial Intelligence (AI) has witnessed rapid resurgence in recent years, due to many innovations in deep learning. Exciting results have been obtained in computer vision (e.g., image classification [Simonyan and Zisserman, 2015; He et al., 2016], detection [Ren et al., 2015; Lin et al., 2017; Zhang et al., 2018a], etc.) as well as natural language processing (NLP) (e.g., [Wen et al., 2016; Li et al., 2017; Zhang et al., 2018b], etc.). Good progress has also been made by researchers in vision-grounded NLP tasks such as image captioning [You et al., 2016; Krishna et al., 2017] and visual question answering [Antol et al., 2015; Malinowski et al., 2015]. Proposed recently, the Visual Dialogue (VD) [Das et al., 2017] task leads to a higher level of interaction between vision and language. In the VD task, a machine conducts natural language dialogues with humans by answering questions grounded in an image. It requires not only reasoning on vision and language, but also generating consistent and natural dialogues.

Existing VD systems can be summarized into two tracks [Das et al., 2017]: generative models and discriminative models. The system adopting the generative model can generate responses while that using the disriminative model only chooses responses from a candidate set. Although discriminative models achieved better recall performance on the benchmark dataset [Das et al., 2017], they are not as applicable as generative models in real world scenarios since candidate responses may not be available. In this work, we focus on the design of generative VD systems for broader usage.

∗Contact Author

One main weakness of existing generative models trained by the maximum likelihood estimation (MLE) method is that they tend to provide frequent and generic responses like 'Don't know'. This happens because the MLE training paradigm latches on to frequent generic responses [Lu et al., 2017]. They may match well with some but poorly for others. There are many possible paths a dialogue may take in the future - penalizing generic poor responses can eliminate candidate dialogue paths and avoids abuse of frequent responses. This helps bridge the large performance gap between generative/discriminative VD systems.

To reach this goal, we propose a novel weighted likelihood estimation (WLE) based training scheme. Specifically, instead of assigning equal weights to each training sample as done in the MLE, we assign a different weight to each training sample. The weight of a training sample is determined by its positive response as well as the negative ones. By incorporating supervision from both positive and negative responses, we enhance answer diversity in the resulting generative model. The proposed training scheme is effective in boosting the VD performance and easy to implement.

Another challenge for VD systems is effective reasoning based on multi-modal inputs. Previous work pre-defined a set of reasoning paths based on multi-modal inputs. The path is specified by a certain sequential processing order, e.g., human queries followed by the dialogue history and then followed by image analysis [Lu et al., 2017]. Such a pre-defined order is not capable of handling different dialogue scenarios, e.g., answering a follow-up question of 'Is there anything else on the table?'. We believe that a good reasoning strategy should determine the processing order by itself. Here, we propose a new reasoning module, where an adaptive reasoning path accommodates different dialogue scenarios automatically.

There are three major contributions of this work. First, an effective training scheme for the generative VD system is proposed, which directly exploits both positive and negative responses using an unprecedented likelihood estimation method. Second, we design an adaptive reasoning scheme with unconstrained attention on multi-modal inputs to accommodate different dialogue scenarios automatically. Third, our results demonstrate the state-of-the-art performance on the VisDial dataset [Das et al., 2017]. Specifically, our model outperforms the best previous generative-model-based method [Wu et al., 2018] by 3.06%, 5.81% and 5.28 with respect to the recall@5, the recall@10 and the mean rank performance metrics, respectively.

2 RelatedWork

Visual dialogue. Different visual dialogue tasks have been examined recently. The VisDial dataset [Das et al., 2017] is collected from free-form human dialogues with a goal to answer questions related to a given image. The GuessWhat task [De Vries et al., 2017] is a guessing game with goaldriven dialogues so as to identify a certain object in a given image by asking yes/no questions. In this work, we focus on the VisDial task. Most previous research on the VisDial task follows the encoder-decoder framework in [Sutskever et al., 2014]. Exploration on encoder models includes late fusion [Das et al., 2017], hierarchical recurrent network [Das et al., 2017], memory network [Das et al., 2017], historyconditioned image attentive encoder (HCIAE) [Lu et al., 2017], and sequential co-attention (CoAtt) [Wu et al., 2018]. Decoder models can be classified into two types: (a) Discriminative decoders rank candidate responses using crossentropy loss [Das et al., 2017] or n-pair loss [Lu et al., 2017]; (b) Generative decoders yield responses using MLE [Das et al., 2017], which can be further combined with adversarial training [Lu et al., 2017; Wu et al., 2018]. The latter involves a discriminator trained on both positive and negative responses, and its discriminative power is then transferred to the generator via auxiliary adversarial training.

Weighted likelihood estimation. Being distinct from previous generative work that uses either MLE or adversarial training, we use WLE and develop a new training scheme for VD systems in this work. WLE has been utilized for different purposes. For example, it was introduced in [Warm, 1989] to remove the first-order bias in MLE. Smaller weights are assigned to outliers for training to reduce the effect of outliers [Ning et al., 2015]. The binary indicator function and the similarity scores are compared for weighting the likelihood in visual question answering (VQA) in [Hu et al., 2018]. We design a novel weighted likelihood remotely related to these concepts, to utilize both positive and negative responses.

Hard example mining. Hard example mining methods are frequently seen in object detection algorithms, where the amount of background samples is much more than the object samples. In [Rowley, 1999], the proposed face detector is trained until convergence on sub-datasets and applied to more data to mine the hard examples alternatively. Online hard example mining is favored by later work [Shrivastava et al., 2016; Lin et al., 2017], where the softmax-based cross entropy loss is used to determine the difficulty of samples. We adopt the concept of sample difficulty and propose a novel way to find hard examples without the preliminary of softmax-based cross entropy.

Multi-modal reasoning. Multi-modal reasoning involves extracting and combining useful information from multimodal inputs. It is widely used in the intersection of vision and language, such as image captioning [Xu et al., 2015] and VQA [Xu and Saenko, 2016]. For the VD task, reasoning can be applied to images (I), questions (Q) and history dialogues (H). In [Lu et al., 2017], the reasoning path adopts the order "Q → H → I". This order is further refined to "Q → I → H → Q" in [Wu et al., 2018]. In the recent arxiv paper [Gan et al., 2019], the reasoning sequence of "Q → I → H " is recurrently occurring to solve complicated problems. Unlike previous work that defines the reasoning path order a priori, we propose an adaptive reasoning scheme with no pre-defined reasoning order.

3 Proposed Generative VD System

In this section, we describe our approach to construct and train the proposed generative visual dialogue system. Following the problem formulation in [Das et al., 2017], the input consists of an image I, a 'ground-truth' dialogue history Ht-1 = ( C ︸︷︷︸ h0 , (Q1, A1) ︸ ︷︷ ︸ h1 , · · · , (Qt-1,At-1) ︸ ︷︷ ︸ ht-1 ) with im-

age caption C and a follow-up question Qt at round t. N candidate responses At = {A1t ,A2t , · · · ,ANt } are provided for both training and testing. Figure 1(a) shows an example fromVisDial [Das et al., 2017].

We adopt the encoder-decoder framework [Sutskever et al., 2014]. Our proposed encoder, which involves an adaptive multi-modal reasoning module without pre-defined order, will be described in details in Sec. 3.1. The generative decoder receives the embedding of the input triplet {I,Ht-1,Qt} from the encoder and outputs a response sequence ˆ At. Our VD system is trained using a novel training scheme with weighted likelihood estimation, which will be described in Sec. 3.2 with details.

E

Figure 2: The adaptive multi-modal reasoning.

3.1 Adaptive Multi-modal Reasoning (AMR)

To conduct reasoning on multi-modal inputs, we first extract image feature FI ∈ RN×H×W by a convolutional neural network, where N is the length of the feature, and H and W are the height and width of the output feature map. The question feature FQ ∈ RN×lQ and history feature FH ∈ RN×lH are obtained by recurrent neural networks, where lQ and lH are the length of the question and the history, respectively.

Our reasoning path consists of two main steps, namely the comprehension step and the exploration step, in a recurrent manner. In the comprehension step, useful information from each input modality is extracted. It is apparent that not all the input information is equally important in the conversation. Attention mechanism is thus useful to extract relevant information. In the exploration step, the relevant information is processed and the following attention direction is determined accordingly. Along the reasoning path, these two steps are performed alternatively.

In [Lu et al., 2017; Wu et al., 2018], the comprehension and exploration steps are merged together. The reasoning scheme focuses on one single input modality at each time and follows a pre-defined reasoning sequence through each input modality. However, this pre-defined order cannot accommodate various dialogue scenarios in real world. For example, a question of "How many people are there in the image?" should yield a short reasoning sequence like

whereas a question of "Is there anything else on the table?" should result in a long reasoning sequence such as

To overcome the drawback of pre-defined reasoning sequence, we propose an adaptive multi-modal reasoning module as illustrated in Figure 2.

Let λ denote any multi-modal feature type (image, question or history), and Fλ ∈ RN×M denote the features to be attended, where M is the number of features. The guided attention operation that paying attention according to the given guide is denoted as fλ = GuidedAtt(Fλ, fg), where fg ∈ RN×1 is the attention guiding feature. The guided attention can be expressed as:

where Wλ, Wg and watt are learnable weights, 1 is a vector with all elements set to 1.

In time step i, the image features FI, the question features FQ and the history features FH are attended separately by their own guided attention blocks. During the comprehension step, the outputs of the guided attention blocks fI,i, fQ,i and fH,i, i.e. the extracted information from each modality, are merged into fQIH,i. During the exploration step, the merged vector is processed in the reasoning RNN block, which generates the new attention guiding feature fg,i to guide the attention in time step i + 1. The final embedding feature E is

where W is learnable weights, imax is the maximum number of recurrent steps.

Through this mechanism, the reasoning RNN block maintains a global view of the multi-modal features and reasons what information should be extracted in the next time step. The information extraction order and subject are therefore determined adaptively along the reasoning path.

3.2 WLE-based Training Scheme

As the discriminative VD models are trained to differentiate positive and negative responses, they perform better on the standard discriminative benchmark. In contrast, the generative visual dialogue models are trained to only maximize the likelihood of positive responses. The MLE loss function is expressed as:

where ppos m denotes the estimated likelihood of the positive response of sample m. There is only one positive response per sample provided for training in the VisDial task. However, there are many possible paths a dialogue may take in the future, the MLE approach therefore favors the frequent and generic responses when the training data is limited [Lu et al., 2017]. In the VisDial task, negative responses are selected from positive responses to other questions, including frequent and generic responses. Incorporating the negative responses to maximize the learning from all available information is thus essential to improve the generative models.

We propose a WLE based training scheme to utilize the negative responses and remedy the bias of MLE. Rather than treating each sample with equal importance, we assign a weight αm to each estimated log-likelihood as:

We can interpret the weighted likelihood as a hard sample mining process. We are inspired by OHEM [Shrivastava et al., 2016] and focal loss [Lin et al., 2017] designed for object detection, where hard samples are mined using their loss values and receive extra attention. Rather than using the preliminary softmax cross entropy loss for discriminative learning, we propose to use likelihood estimation to mine the hard samples. If the current model cannot predict the likelihood for a sample well, it indicates that this sample is hard for the model. Then we should increase the weight for this hard sample and vice versa.

Given both positive and negative responses for training, we propose to assign weights as:

where pneg m,n denotes the n-th negative response of sample m, τ and γ are hyper-parameters to shape the weights.

We can also view the proposed loss function as a ranking loss. We assign a weight to a sample by comparing the estimated likelihood of its positive and negative responses. βm,n measures the relative distance of likelihood between the positive response and the n-th negative response of sample m. If the likelihood of a positive response is low comparing to the negative responses, we should penalize more by increasing the weight for this sample. If the estimated likelihood of a positive sample is already very high, we should lower its weight to reduce the penalization.

4 Experiments

4.1 Dataset

We evaluate our proposed model on the VisDial dataset [Das et al., 2017]. In VisDial v0.9, on which most previous work has benchmarked, there are in total 83k and 40k dialogues on COCO-train and COCO-val images, respectively. We follow the methodology in [Lu et al., 2017] and split the data into 82k for train, 1k for val and 40k for test. In the new version VisDial v1.0, which was used for the Visual Dialog Challenge 2018, train consists of the previous 123k images and corresponding dialogues. 2k and 8k images with dialogues are collected for val and test, respectively.

Each question is supplemented with 100 candidate responses, among which only one is the human response for this question. Following the evaluation protocol in [Das et al., 2017], we rank the 100 candidate responses by their estimated likelihood and evaluate the models using standard retrieval metrics: (1) mean rank of the human response, (2) recall rate of the human response in top-k ranked responses for k = 1, 5, 10, (3) mean reciprocal rank (MRR) of the human response, (4) normalized discounted cumulative gain (NDCG) of all correct responses (only available for v1.0).

4.2 Implementation Details

We follow the procedures in [Lu et al., 2017] to pre-process the data. The captions, questions and answers are truncated at 24, 16 and 8 words for VisDial v0.9, and 40, 20 and 20 words for VisDial v1.0. Vocabularies are built afterwards from the words that occur at least five times in train. We use 512D word embeddings, which are trained from scratch and shared by question, dialogue history and decoder LSTMs.

For a fair comparison with previous work, we adopt the simple LSTM decoder with a softmax output which models the likelihood of the next word given the embedding feature and previous generated sequence. We also set all LSTMs to have single layer with 512D hidden state for consistency with other works. We extract image features from pre-trained CNN models (VGG [Simonyan and Zisserman, 2015] for VisDial v0.9, ResNet [He et al., 2016] or bottom-up features [Anderson et al., 2018] for VisDial v1.0), and train the rest of our model from scratch. We use the Adam optimizer with the base learning rate of 4 × 10-4 .

4.3 Experiments Results and Analysis Baselines

We compare our proposed model to several baselines and the state-of-the-art generative models. In [Das et al., 2017], three types of encoders are introduced. Late Fusion (LF) extracts features from each input separately and fuses them in the later stage. Hierarchical Recurrent Encoder (HRE) uses hierarchical recurrent encoder for dialogue history and HREA adds attention to the dialogue history on top of the hierarchical recurrent encoder. Memory Network (MN) uses memery bank to store the dialogue history and find corresponding memory to answer the question. History-Conditioned Image Attentive Encoder (HCIAE) is proposed in [Lu et al., 2017] to attend on image and dialogue history and trained with generative adversarial training (GAN). Another concurrent work with GAN [Wu et al., 2018] proposes a co-attention model (CoAtt) that attends to question, image and dialogue history. FlipDial [Massiceti et al., 2018] uses VAE for sequence generation. We also compare to a neural module network approach Coref [Kottur et al., 2018] in which only the performance with ResNet [He et al., 2016] backbone is reported. ReDAN [Gan et al., 2019] is recently proposed method which involves a multi-step reasoning path with pre-defined order.

Results on VisDial v0.9

Table 1 compares ours results to other reported generative baselines. Our model performs the best on most of the evaluation metrics. Comparing to HCIAE [Lu et al., 2017], our model shows comparable performance on R@1, and outperforms on MRR, R@5, R@10 and mean rank by 1.47%, 3.47%, 6%, 5.08, respectively. Our model also outperforms Time step i = 1

Time step i = 2

Figure 5: Visualization of image attention heatmaps for different questions and reasoning steps. Regions of attention are highlighted in blue.

CoAtt [Wu et al., 2018], which achieved the previous best results for generative models. Our result surpass it with large margins on R@5, R@10 and mean rank by 3.06%, 5.81% and 5.28, respectively.

While our model demonstrates remarkable improvement on R@5, R@10 and mean rank, MRR shows moderate gain while R@1 is slightly behind. We attribute this to the fact that there could be more than one correct response among the candidates while only one is provided as the correct answer. As demonstrated by the examples of top-10 responses in Figure 3, our model is capable of ranking multiple correct answers to higher places. However, the single human answer is not necessarily ranked the 1st, thus greatly affecting R@1.

Results on VisDial v1.0

In the Visual Dialog Challenge 2018, all correct responses in test are annotated by humans and considered in the evaluation. Figure 4 represents the top-10 results. Our model, as the only generative model in the top-10, ranked as the 6th among those discriminative models. It also verifies our claim that our low R@1 score on v0.9 is because the evaluation only considers the human response but ignore all other correct responses. We used ResNet features for the challenge. Since ReDAN only reports its generative performance on VisDial v1.0 val with bottom-up features, we also present our results using the same setting in Table 2. We list the results of previous work in Table 2 as reported in [Gan et al., 2019]. Similar to the results on VisDial v0.9, our proposed method outperforms previous methods on MRR, R@5, R@10 and Mean.

Ablation Study

Our model contains two main novel components, namely the adaptive multi-modal reasoning module and the WLE based training scheme. To verify the contribution of each component, we compare the following models : (a) HCIAE-MLE is the HCIAE model trained via MLE; (b) HCIAE-GAN is the HCIAE model trained via MLE and GAN; (c) HCIAEWLE is the HCIAE model trained via WLE; (d) AMR-MLE is our AMR model trained via MLE; (e) AMR-WLE is our Do you think it is sunny outside ? Yeah A couple of men are standing in the woods HCIAE-WLE performs better on all metrics. Specifically, the improvement on the HCIAE model by WLE is more than twice of that by GAN on R@10 (6.35 vs. 2.31) and mean rank (6.08 vs. 1.78). Our proposed training scheme is therefore also compatible and effective with other encoders.

Qualitative Results

Examples of image attention heatmaps are visualized in Figure 5, which demonstrate the adaptive reasoning focuses for different questions and reasoning time steps. For example, for the second question, the attention on image was first at a large area of background, then moved to more focused region to answer the question 'any buildings'.

Figure 6 shows some qualitative results on test. Our generative model is able to generate more non-generic answers. As evidently shown in the comparison between MLE and WLE, the WLE results are more specific and human-like.

5 Conclusion

In this work, we have presented a novel generative visual dialogue system. It involves an adaptive reasoning module for