Exemplar Guided Neural Dialogue Generation

1 Introduction

Sequence-to-sequence (SEQ2SEQ) learning [Bahdanau et al., 2015; Sutskever et al., 2014; Cho et al., 2014] has being a state-of-the-art neural network framework for response generation. It treats dialogue generation as a source to target sequence translation problem, where an encoder network [Cho et al., 2014] encodes the context into a vector representing the semantics of the context, and then a decoder network generates the response word-by-word, conditioned on the context vector. Though e�ective, common wisdom suggests that these models are plagued by the notorious problem of dull, safe responses [Li et al., 2016; Zhang et al., 2018].

� Work done at Data Science Lab, JD.com.

This phenomenon occurs partially because that existing models attempt to generate responses for all those conversation contexts based solely on its learnt model parameters [Pandey et al., 2018]. Since human dialogues typically conducted in an open-ended and highly subjective way, capturing all information required to generate responses barely by the model parameters is not necessarily adequate.

Fortunately, exemplary conversations, which can be exploited to improve the response generation, are usually embodied in the training set. As observed in Figure 1, by referring to the exemplar expression "I like in my free time", the user composes an appropriate utterance "I like painting in my free time" responding to the post message "Do you like art?". Such closely related samples provide the model with explicit referable exemplary information that benefit the dialogue generation when responding to a given input message. Based on this observation, it is reasonable to extend the neural dialogue generation model to explicitly take into consideration such relevant exemplary data from the full training set. To augment neural dialogue generation with exemplar conversations, similar examples are first retrieved from training data, and the exemplar responses are then fed into the decoder as exemplar vectors to generate the response [Pandey et al., 2018].

Nevertheless, we observe that, e�ectively fusing such exemplary information into dialogue generation is not that straightforward, but still rather challenging: although the retrieved conversations are similar literally, some of them are topic-unrelated with the given context. Figure 1 shows two exemplar conversations. Both of them are similar to the given context "Do you like art?" literally. However, the upper one discusses "soccer", while the given context and the lower conversation talk about "art". Those topic-unrelated exemplar responses hinder the neural dialogue generation model understanding the exact topic of the conversation, and the model may refer to inappropriate exemplar responses during generation. What's more, simply encoding exemplar responses into hidden vectors further aggravates the inaccurate dialogue topic problem.

Viewing this, in this paper, we propose an Exemplar guided Neural Dialogue generation model-End, where exemplar responses are retrieved in terms of both the text similarity and the topic proximity through a two-stage exemplar retrieval model. In the first stage, a small subset of conversations is retrieved from a training set given a dialogue context. These candidate exemplars are then finely ranked regarding the topical proximity to choose the best-matched exemplar response and guide the dialogue response generation. To further enhance the neural dialogue generation model leveraging the exemplar response and the conversation topics e�ectively, we introduce a multi-source sampling mechanism during decoding, where the response word can be drawn from the vocabulary embedding space and exemplary collections. The exemplar response is utilized as a soft response template, which can be viewed as local exemplary signals, whereas the dialogue topics serve as global exemplary semantics.

We evaluate the proposed exemplar guided neural dialogue generation model on a real-life conversation dataset. Our experiments reveal that the proposed approach e�ectively exploits the exemplary information and achieves significant improvement over the strong baselines.

2 Exemplar Guided Neural Dialogue Generation

In this work, we design a neural dialogue system, in which the response generation is guided by the exemplary conversations. Unlike the conventional neural dialogue generation models, the proposed model maintain and actively exploit the training corpus during response generation. As illustrated in Figure 2, End mainly consists of two components: (a) Given an input message x, an exemplary conversation retriever distills the closely related conversations regarding both the text similarity and the topic proximity. (b) An encoder-decoder model generates the final response y under the guidance of the recognized exemplary information.

2.1 Exemplary Conversation Retriever

The proposed exemplary conversation retriever first retrieves a small subset of conversations from a training set and then refine the retrieved conversations based on topic proximity to ameliorate the noisy exemplar issue.

First-round Exemplar Retrieval. Given a context, the related conversations are gathered by the exemplar responses retriever, which recalls the exemplar responses from the training set based on the semantic distance between the given context and the candidate dialogue contexts.

With a huge number of training examples, it is prohibitively expensive at run time to calculate the proximity to the query context iteratively over the whole set of exemplars. We hence index the whole training set in term space. For a query context x, the top-N potential exemplar contextresponse pairs (c (i) ; r (i) ) N i=1 are retrieved by BM25 [Robertson and Zaragoza, 2009]. Note that other sophisticated retrieval models can also be applied in the first round retrieving, e.g., locality sensitive hashing [Indyk and Motwani, 1998].

As aforementioned, retrieving the exemplar responses solely based on the superficial context words is risky, since the noisy exemplars will mislead the model and the resultant irrelevant response usually makes people end the conversation quickly. We therefore finely rank the set of retrieved exemplars and choose the best-matched exemplar response, based on topical proximity. We first introduce the variational topic inference and then elaborate the exemplars reranking.

Latent Topic Inference. We introduce the neural variational topic model [Miao et al., 2017] to approximate the conversation topics of a given dialogue d. d is composed of a context-response pair (x, y). Following Miao et al. [2017], we adopt an inference network to parameterize the latent topic distribution � and a multinomial softmax generative model to reconstruct the conversation based on the topic vectors from the latent topic distribution. More concretely, a latent variable � is parameterized by an inference network P(�j�pri(x); �pri(x)), which approximates the posterior Q(�j�pos(d); �pos(d)). P and Q are conditioned on a draw from a Gaussian distribution. Outputs of functions � and � are parameters of the Gaussian distribution, which are computed using multilayer perceptrons (MLP). We use the reparameterization trick [Kingma and Welling, 2014] to guarantee di�erentiability when sampling from Gaussian distributions. The topic distribution � is then built using � by � = softmax(Linear(�)). Given �, the dialogue d is reconstructed by computing the marginal likelihood:

where the log-likelihood of a word wi can be factorized as:

zi is the topic assignment and � is the topics-words similarity matrix. We further introduce z 2 R M�H as the topic word embeddings, � 2 R K�H as the topic embeddings and generate the topic-words similarity matrix � by: �k = softmax(z � � T k ), where K represents the topic number, M denotes the number of topic words and H stands for the embedding size.

Exemplar Response Refining. To refine the retrieved exemplar responses using the latent topics and make the dialogue generation robust to the noisy exemplars, we finely rank the set of retrieved exemplars and choose the best-matched exemplar response, based on topical proximity. For each candidate exemplar context-response (c (i) ; r (i) ), the ranking function is as follows:

where �i is the topic proportion of the exemplar contextresponse pair (c (i) ; r (i) ), computed through the latent topic inference network, and �x is the topic proportion of the query context. The retrieved exemplar response with the highest ranking score-rt, will be adopted to guide the response generation.

2.2 Exemplar Guided Response Generation

Given an input sequence fw1; w2; :::; wng, we adopt a bidirectional RNN to transform the discrete tokens into hidden representations. The final hidden states in two directions are then concatenated to forma sentence representation h = [ �! h T n ; � h T n ]. For the input context x and the retrieved exemplar response r, they are both in the form of sequences, and will be transformed into their corresponding distributed representations hx and hr through separate bidirectional LSTMs respectively.

The decoder generates the response sequentially through a forward RNN. For the j-th word, the scoring function relies on the decoding hidden state s j and the exemplary embeddings which consist of the exemplar response rt and the topic embedding t computed as t = � �. The architecture of the decoder is shown in Figure 3.

Vanilla Decoder. The vanilla decoder simply generates the response word yj conditioned on the context x and the previous generated words y[1: j�1]. Then, the probability of the response y is as follows:

where Ty is the length of response y and s j denotes a combination of the source context information and the recurrent hidden state up to time step j. The vanilla decoder does not exploit any exemplary information. All required information are conveyed through the hidden s. Relying merely on the context hidden states, the model often gets in trouble for generating appropriate responses.

Decoding with Exemplar Response. Conventional practice exploiting the exemplar response simply encodes the exemplars as hidden vectors, which may lead to the loss of exemplary information. We hence employ the exemplar response as a soft language template, allowing the response word to be drawn from the exemplary collections. As shown in Figure 3.(a), we integrate the exemplar response rt into the response generation. In decoding, the generation probability p(y j) can be defined as:

where p�V (yj) and p�(yj) are the probabilities of generating yj from the conventional vocabulary �V and exemplar response rt, respectively, and are computed as:

where Z1 = P w2�V e ��V (w) + P w2rt e ��(w) is the normalization term. ��V and �� are the scoring functions and r m t stands for the m-th word in rt. ��V and ��(r m t ) are defined as:

where �V and �� are non-linear transformation functions, like multi-layer perceptrons, to project the input into the scoring vector. hrt is the hidden representation of exemplar response rt, and w is a one-hot indicator vector of word w.

Decoding with Dialogue Topics. To further provide the dialogue response generation with global topical exemplary semantics, we extend the response generation by sampling response words from the topic words vocabulary ��. When generating a response word, the model predicts the word probabilities by referring to both the conventional vocabulary and the topic words vocabulary, as illustrated in Figure 3.(b). Then, the generation probability p(y j) can be defined as:

where p�� (yj) is defined by:

Z2 = P w2�V e ��V (w) + P w2�� e ��� (w) is the normalization term. ��� (w) is computed by:

where �� is a non-linear transformation function, like multilayer perceptrons, to project the input s j and t into the scoring vectors. w is a one-hot indicator vector of word w. p�V (yj = w) is formulated similarly as in Eq.(6).

Exemplar-EnhancedGating. In order to dynamically control the e�ects of the exemplary information in the process of dialogue response generation, we further introduce a gating mechanism for the scoring functions. We utilize the exemplary embeddings, including the exemplar response hrt and topic embedding t, together with the decoding hidden state s j, to perform gating. The scoring functions are updated as gated scoring functions:

where GV , G� and G� are the gating functions, which can be implemented as simple as a sigmoid function or as a gated recurrent unit. At each time step, the gating functions control whether or not the next response word is generated, referring to the exemplar response and topic information. When G(1j = 0), it indicates that the decoder hidden state s j is informative enough to score the next response words, while G(1j = 1) denotes that the exemplary information should be taken more into account.

Exemplar Guided Decoder. The full version of the exemplar guided exemplar decoder jointly utilizes all the proposed mechanisms to generate the final response. When generating a word yj, both the exemplar response and topic words are integrated through gated multi-source sampling mechanisms. The generation probability of y j can be finalized as:

and Z = P w2�V e ��V (w) + P w2rt e ��(w) + P w2�� e ��� (w) is used to normalize the scores. Figure 3 details the j-th word generation in the proposed decoder.

Optimizing. End are trained to maximize the generation likelihood of the given parallel corpus as well as the variational lower bound of the latent topic inference:

where the first term is the conventional response generation objective, the second term is the dialogue generation objective in latent topic inference, and the third term is the KL divergence between two Gaussian distributions.

3 Experiments

Dataset. To validate our model's e�ectiveness, we construct an open-domain conversation corpus spanning over several public available dialogue dataset, including a movie discussions dataset collected from Reddit [Dodge et al., 2015], and a Ubuntu technical corpus [Lowe et al., 2015] discussing about the usage of Ubuntu. These datasets are widely used in dialogue researches [Pandey et al., 2018]. 57,402 context-response pairs are sampled for training, 3,000 for validation and 3,000 for testing.

Hyper Parameters and Reproducibility. Our model is implemented using ParlAI [Miller et al., 2017]. We truncate all context utterances to length 100 and response utterance to length 50. We take the most frequent 20,000 words as conventional vocabulary. Regarding model implementations, the RNNs in the encoder and the decoder utilize 2-layer LSTM structures with 256 hidden cells for each layer. The latent variable size is set to 64. The size of latent topics is set to 10. The dimensions of word embedding and topic embedding matrix are set to 300. Top-10 candidate exemplar responses are retrieved by the exemplar responses retriever in the first round retrieving. The Adam [Kingma and Ba, 2014] optimizer with a learning rate of 0.001 is used to train the models. We use early stopping with log-likelihood on the validation set as the stopping criteria.

Baselines. We compare the proposed End with the following state-of-the-art baselines. 1) SEQ2SEQ+Attention: Attention-based sequence-to-sequence model [Bahdanau et al., 2015] is a representative baseline. It is denoted as SEQ2SEQ hereafter; 2) CVAE: Latent variable conversational model [Clark and Cao, 2017; Zhao et al., 2017] is a derivative of the SEQ2SEQ model in which it incorporates a latent variable at the sentence-level to inject stochasticity and diversity; 3) LAED: A recurrent encoder-decoder model [Zhao et al., 2018] using discrete latent actions for interpretable neural dialogue generation; 4) EED: A conversation model [Pandey et al., 2018] that utilize similar examples from training data to generate responses; 5) CopyNet: An attention-based sequence-to-sequence model augmented with copy mechanism [Gu et al., 2016]; 6) TAS2S: TAS2S [Xing et al., 2017] incorporates the topic information into the response generation, where the topics are learned from a separate LDA model to enrich the context, resulting with more informative and interesting responses.

Automatic Evaluation Metrics. The BLEU [Papineni et al., 2002] metric is employed to measure the response quality. Besides, in order to evaluate the semantic relevance between the generated response and the ground-truth response, we also adopted the embedding-based similarity metrics proposed by Liu et al. [2016]: Embedding Average (Ave.), Embedding Extrema (Ext.) and Embedding Greedy (Gre.). To measure informativeness and diversity of the response, we also exploited the Distinct-n metrics (n=f1,2,3g).

Overall Performance. In Table 1, we compare the results of our model with all the baselines in terms of both the relevance metrics and the informativeness metrics. Overall, we observe that our model exceeds all the comparison models on automatic evaluation metrics.

For relevance metrics, CVAE, LAED and TAS2S surpass the original attention-based SEQ2SEQ baseline regarding the BLEU score and embedding-based evaluation, which is consistent with the reports in Xing et al. [2017]. It indicates that both the latent variable and the topic information slightly enable SEQ2SEQ generating more appropriate responses. EED exhibits competitive BLEU score improvement among baselines, implying that the exemplar response is helpful to promote the response relevance. CopyNet also improves the embedding-based metrics a lot compared with SEQ2SEQ, owing to its ability to copy words from the context. The improvements of End over SEQ2SEQ are even larger than the baseline models, which demonstrates the benefits of exploiting the exemplary information from the training corpus. As the topic information is automatically inferred during response generation, the error accumulation problem is reduced, comparing with exploiting the fixed pretrained topic information as in Xing et al. [2017]. In terms of informativeness, CVAE, LAED, CopyNet and TAS2S also achieve better performances comparing to SEQ2SEQ, whereas our model presents much larger improvements in Distinct-f1,2,3g metrics. It implies that, under the guidance of exemplary information, our model is more adept at generating diverse dialogue responses. We also conducted significance tests with t-test for relevance metrics and Sign-test [Dixon and Mood, 1946] for Distinct metrics. End significantly outperforms the baselines on the majority of metrics with p-value < 0:01.

Model Ablation. To examine the e�ectiveness of the exemplar response and topic information in response generation, we conducted model ablations by removing particular modules from End. As shown in Table 2, we observe that without either the exemplar response or conversation topics, the performance drops rapidly with respect to all the evaluation metrics. It verifies the e�ectiveness of decoding with the exemplary information. Note that the performance drops when the topic information excludes from the exemplar decoding, a�rming that the conversation topics are helpful to refine the retrieved exemplars for response generation. In line (3) of Table 2, when both the exemplar response and conversation topics together incorporated in response generation, compared to the decoding with either the exemplar response or conversation topics, the model obtains much better performance. Finally, the exemplar-enhanced gating mechanism further improves the performance and achieves the best results (line (4) in Table 2).

Human Evaluation. We also carried out the human study through comparisons between our model and the baselines, following Wang et al. [2018]. For each case, given a contextresponse pair, two generated responses were provided, one is from our model and the other is from the comparison model. We randomly selected 500 samples from the test set. Three well-educated students were invited to rate which one is better considering four factors: context relevance, logical consistency, fluency and informativeness. Notice that cases with di�erent evaluations are marked as "tie". Table 3 summarizes the results of subjective evaluation. The kappa scores indicate that the annotators came to a fair agreement in the judgment. Not surprisingly, as shown in Table 3, End consistently outperforms all the comparison systems. However, we notice that CVAE and LAED exhibit insignificant improvements comparing with SEQ2SEQ. We analyzed the bad cases and found that CVAE still su�ers from the general, meaningless responses. Meanwhile, though LAED generates more informative responses, some responses are irrelevant to the given context. Augmented with the exemplar response and inferred latent topics, End introduces a competitive boost in response quality, which is in line with the automatic evaluation, confirming the superior performance of the proposed method.

Case Study. Table 4 lists several responses generated by End and the baselines. The exemplar responses and the top-5 inferred topic words are also presented. The exemplar responses and discovered topic words contain informative contents that are utilized by the End model for generating responses. In the first example (line 1 in Table 4), we notice that the topic words "trains" and "travel" are decoded into the response. This is in concert with our intuition that latent topic inference helps to provide End with informative topical information. As for example 2 in Table 4, the exemplar response provides the model with a soft template "download from " and influences how to say it. Regarding line 3 in Table 4, End benefits from both the inferred topic words and the exemplar response, and composes an appropriate phrase "watch movies with you" by consulting the topic words "movies, watch" and the exemplar phrase "with you". In general, we found that End is able to e�ectively fuse such exemplary information into the dialogue response generation.

4 RelatedWork

To improve the neural dialogue systems, prior art typically focuses its attention on elaborately exploiting the given conversation for response generation, by using latent variables [Serban et al., 2017; Clark and Cao, 2017; Zhao et al., 2017; Zhao et al., 2018], hierarchical history modeling [Serban et al., 2016; Chen et al., 2018], input-dependent parameterization [Cai et al., 2019] or predicting keywords from context [Yao et al., 2017; Wang et al., 2018].

In contrast to the above models, our model takes into account more information from the whole training set than a current dialogue context for generating the final responses. Pandey et al. [2018] encoded the exemplar response into a hidden vector, which may lead to the loss of information, while we utilize the exemplar response as a template through a copying mechanism [Gu et al., 2016]. To ameliorate the problem of noisy exemplars, we also refine the retrieved exemplar responses using the inferred latent topics. While the principal idea of both the papers remains similar, the di�erence lies in the mechanism of gathering and incorporating the retrieved data. Xing et al. [2017] incorporated topic information into the SEQ2SEQ dialogue response generation.

Wang et al. [2017] biased the generation process with a topic restriction. However, their topic information is obtained through pre-trained models. Yao et al.; Wang et al. [2017; 2018] leveraged the predicted keywords to boost the response informativeness, which does not involve topic modeling actually. While in our model, the latent topics are automatically inferred from the given dialogue and the model is trained within a unified framework in an end-to-end fashion. Another di�erence is that, they only utilized the topic words to guide the response generation, while we enhance the response generation with both exemplar responses and latent topics.

5 Conclusion

In this work, we present End-a novel neural dialogue generation model which considers not only a given conversation context, but also a set of relevant exemplary conversations from the training corpus in the process of response generation. To provide the dialogue model with beneficial exemplars, the proposed approach adopts a two-stage exemplar retrieval model: in the first stage, a small subset of conversations is retrieved from a training set given a dialogue context; these candidate exemplars are then refined regarding the topical proximity to choose the best-matched exemplar response. To e�ectively fuse such exemplary information into dialogue response generation, we further introduce a multisource sampling mechanism to provide the dialogue model with both local exemplary semantics and global topical guidance during response decoding. Extensive experiments show that the proposed model outperforms the state-of-the-art baselines and is capable of generating more informative and relevant responses.

Acknowledgements

We would like to thank all the reviewers for their insightful and valuable comments and suggestions. Hongshen Chen and Xiaofang Zhao are the corresponding authors.