EmoElicitor: An Open Domain Response Generation Model with User Emotional Reaction Awareness

1 Introduction

Emotional interaction is a key factor in interpersonal communication and has become a crucial concern in building humanlike dialogue agents [Picard, 1997]. Ample evidence [Partala and Surakka, 2004; Prendinger and Ishizuka, 2005] has shown that agents capable of expressing emotions can significantly improve user satisfaction during human-computer interactions. Emotional response generation (ERG) is an emotional interaction task for generating appropriate conversational responses conditioned on a given emotion. Early studies [Skowron, 2009] manually designed rules to select the desired emotional responses from a corpus. More recently, great achievements have been witnessed along this line of research due to the availability of large-scale dialogue data [Zhou and Wang, 2018; Rashkin et al., 2019] and the development of Seq2Seq models [Sutskever et al., 2014; Zhou et al., 2018; Song et al., 2019; Zhong et al., 2019].

Although the previous methods have achieved promising results, these models have attempted only to control the emotion of the agent's response. Meanwhile, the feelings of the user during the interaction, which are actually the ultimate concern when designing a dialogue agent, are neglected. In assure users and that the emotion label Happiness is provided to two agents. The traditional ERG method (Agent A) can generate the topic-relevant (cat) response with the right emotion. However, Agent A fails to comfort the user because it does not take the user's emotional reaction into account. By contrast, Agent B can elicit the emotion of Happiness from the user because it considers the perspective of the user and generates a more appropriate response based not only on the backward context but also on the user's likely next utterance, with the corresponding emotion.

In this paper, we focus on the task of open domain response generation for emotion elicitation (RGEE), in which, given the backward context and a desired emotional reaction (considering the next-round utterance and its emotional reaction label), the objective is to elicit a topic-coherent response that can elicit the specific desired emotion. The RGEE task is fundamentally different from the ERG task. (i) Different goals: The goal of ERG is to generate a response that exhibits a specific emotion, whereas RGEE focuses on eliciting a specific emotion to enhance the interactiveness of chatting. Thus, RGEE requires responding from a user-oriented perspective and is more proactive than traditional ERG. (ii) Different inputs: The emotion labels are conceptually different for the two tasks. Furthermore, during the training phase, RGEE considers not only the backward context but also the next-round utterance of the user, which has rarely been considered in previous dialogue generation models.

RGEE facilitates the construction of a more believable and human-like chat agent that can conduct empathetic interactions [Rashkin et al., 2019] or meet a user's emotional needs [Picard and Klein, 2001]. However, RGEE is a highly challenging task since the generated response is not directly associated with the emotional reaction label. Moreover, although incorporating user's next-round utterance during training facilitates the generation of more coherent responses, the nextround utterance is not available during the inference phase.

To tackle these challenges, we propose a variational model EmoElictor, which is built upon a pre-trained language model [Yang et al., 2019], and can capture the relationship between emotional reactions and responses. The original single latent variable is decomposed into sequential ones at each time step during the generation, which helps to enhance the topic coherence and emotional consistency of the response.

Our three main contributions can be summarized as follows: (i) We formulate the RGEE problem as the problem of generating a response that can elicit a specific emotion from a user conditioned on the backward context and next-round utterance. (ii) We propose a variational model EmoElicitor which, to best of our knowledge, is the first to leverage sequential latent variables to capture context information and guide response generation with the help of pre-trained language model. (iii) We construct a large-scale dataset for the RGEE task 1 . Experimental results show that our model consistently outperforms strong baseline methods.

2 RelatedWork

Emotion-aware dialogue systems have become an emerging area of research in recent years. Zhou et al. [2018] first incorporated emotional factors into a dialogue generation task using an end-to-end neural learning framework. Zhong et al. [2019] considered the VAD affect model and the effects of negators and intensifiers via an attention mechanism in conversation modeling. Zhou and Wang [2018] proposed a reinforced CVAE-based model called Mojitalk, which could generate responses based on emojis. Rashkin et al. [2019] focused on empathethic dialogue generation, in which each conversation contained only one emotion label.

Most previous studies, however, have considered only the emotion of the agent's response while neglecting the user's emotional reaction. Lubis et al. [2018; 2019] generated responses that could elicit positive emotions. Hasegawa et al. [2013] leveraged a statistical machine translation model to generate responses that could elicit predicted emotions from users. In contrast to the above two methods, our model not only utilizes finer-grained emoji labels but also considers the user's next-round utterance to generate more topic-coherent and emotionally consistent responses.

For a given context, there could be multiple appropriate responses; hence, response generation is known to be a oneto-many problem. Variational autoencoders (VAEs) are one of the most successful types of models for solving such problems [Serban et al., 2017]. However, traditional VAE-based models utilize only a single latent variable to encode an entire response sequence. Our work is inspired by variational recurrent neural networks (VRNNs) [Chung et al., 2015], which perform variational inference at every time step during decoding. Variational models suffer from posterior collapse issues; however, these can be alleviated by means of KullbackLeibler (KL) annealing and auxiliary loss.

1 https://github.com/neuChatbotDS/emoElicitorData

3 Our Approach

3.1 ProblemFormulation

Given a dialogue context C and an emotional reaction label e for the next-turn speaker, we aim to generate a response Y that not only is coherent with the context C but also can elicit the desired emotion e from the next-turn speaker.

The dialogue context is C = {U b 1 , U b 2 , U a 3 , . . . ,U b i }, where U b i denotes the utterance of person b in round i. The response is Y = {U a i+1 }. The emotional reaction e is the emotion label of the next-round utterance, U b i+2 ; hence, Ue denotes the next-round U b i+2 .

3.2 CVAE

Our method is built upon conditional VAE (CVAE) [Kingma and Welling, 2014], for where the generation of response y is based on the given context C, the next-round utterance Ue, the emotional reaction label e, and a latent variable z that is intended to capture the distribution of responses.

where KL is the Kullback-Leibler divergence, p(Y |z, C, e) is a decoder that generates Y from the latent variable z, conditional context C and emotional reaction e; pφ(z|C, e) is the prior model used to sample z from the prior distribution; qθ(z|C, Y,Ue) is the posterior network used to approximate the true posterior distribution of the latent variable z; θ and φ are the parameters of the model. In the training phase, the latent variable z sampled from the posterior model is used to generate a response p(Y |z, C, e) given conditions C and e. In the inference phase, only the latent variable z sampled from the prior model is utilized to generate a response p(Y |z, C, e).

Inspired by variational recurrent neural network, our model uses a latent variable zt in each time step to generate a response as pφ(Y |z, C, e) = ∏t p (Yt|Y<t, zt, C, e).

3.3 Model Framework

Input Representation

The inputs to our multi-turn dyadic dialogue model are the conversation context C, the response Y , and the next-round utterance Ue with emotional reaction label e. All input texts are tokenized using SentencePiece 2 . To capture fine-grained and realistic sentiment labels, we regard emojis as the labels of utterances, and these emojis are treated as plain text during processing. The input embedding for each token includes a word embedding, a type embedding and a position embedding, as visualized in Figure 3.

2 https://github.com/google/sentencepiece

ࢁࢋ

(i) The input representation is the concatenation of C, Y and Ue with a special beginning token [s] and end token [/s]. For the context C, we concatenate all the utterances in C with a special token [SEP]. We add an auxiliary emoji vocabulary for emojis. (ii) Type embeddings are employed to differentiate the speakers (e.g. A and B) and the emotion label in the dyadic conversation data. Suppose that speaker A's utterance elicits an emotion from speaker B. Thus, type embeddings of Y and Ue are EA and EB, respectively. Ee denotes emotional reaction label type embedding. (iii) The position embeddings are consistent with those of the pre-trained model.

Our input representation is compatible with the Transformer-based pre-trained language models that include token, segment and position embeddings. Note that we replace segment embedding with type embedding, which is consistent with the features of dyadic conversations and fine-tune the type embeddings on the training dataset.

Framework

Figure 2 shows the overall framework of our proposed model, which is a combination of a pre-trained language model and a variational model. The backbone of our infrastructure is inspired by [Lample and Conneau, 2019], a technology that flexibly supports bi-directional encoding and uni-directional decoding through specific self-attention masks. To better capture the backward and forward context representations, we use a bi-directional attention Transformer to model the context C and the emotional reaction label e. A uni-directional attention Transformer is leveraged to model the response Y

to be generated. The encoder and decoder Transformers share the same set of weights.

Specifically, the input and output are respectively denoted by Ein and Eout for the encoder, and Din and Dout for the decoder. All inputs are sequences of tokens, including token embeddings (TOK), type embeddings (TY P), and position embeddings (POS):

where C; e denotes the concatenation of C and e. The encoder can attend its own tokens (e.g., Ein i (C; e) can attend arbitrary Ein j (C; e)), and each token in the decoder can attend only those ahead of it.

where Ein (C; e);Din (Y ) denotes the concatenation of Ein (C; e) and Din (Y ); Dout j (Y ) is the final output of the decoder when taking response Y as input at time step j; Woo ∈ Rdmodel×dvocab and bo ∈ Rdvocab are parameters of the pre-trained model, with dvocab being the vocabulary size, which is the same as the size of the original vocabulary because our responses do not contain emojis; and Pj is the probability distribution of the word to be generated at time step j.

3.4 EmoElicitor

Traditional CVAE-based methods employ only one latent variable z, which make it difficult to model the distribution of the response p(y|z, c). To more effectively use latent variables to capture the relationship between emotional reactions and responses, we decompose z into sequential variables zt at each time step t during the generation process. Moreover, we add an auxiliary loss to avoid posterior collapse.

EmoElicitor Model Details

On the basis of a CVAE and a pre-trained language model, we build EmoElicitor which leverages the context C, the response Y and the next utterance Ue to guide response generation at every time step t. The posterior net encodes C, Y and Ue. Since we assume that zt follows an isotropic Gaussian distribution N (µ, σ2 I) , we have

where Ein (C; Y ;Ue) is the input representation for the bidirectional attention Transformer, which concatenates the context C, the response Y and the next utterance Ue as described in Section 3.3. Note that the position embeddings are also sequential. Eout (Y ) is the bi-directional attention output representation of the response, which is a sequential representation Eout t (Y ) of length n - 1 because the beginningof-sentence token [s] is not needed in the output representation of response Y for the computation of the posterior latent variables. zt denotes the posterior latent variable at time step t. Wq ∈ Rdmodel×dz and bq ∈ Rdz are weight parameters, where dz is the dimensionality of the latent variable and dmodel is the dimensionality of the pre-trained model output at each time step t. The prior net encodes the context C and the reaction emotion e. Similarly, we have

where Ein (C; e) is the input representation for the bidirectional attention Transformer, which concatenates the context C and the next-utterance emotion e. Din (Y ) is the input representation for the uni-directional attention Transformer, which attends those tokens ahead of it. Dout (Y ) is the uni-directional attention output representation of the response, which is a sequential representation Eout t (Y ) of length n - 1, because the response Y includes an end-ofsentence token [/s] that is not needed during the generation process. MLPp is a multi-layer perceptron.

We incorporate time step latent zt into the pre-trained model's output by a combination layer to predict Yt by computing p (Yt|Y<t, zt, C, e).

where Wg ∈ R(dmodel+dz)×dz is the weight parameter used to combine zt+1 and Dout t (Y ) into Gt. Wo ∈ R(dmodel+dz)×dvocab includes Wo1 ∈ Rdmodel×dvocab and Wo2 ∈ Rdz×dvocab : the first part loads the parameters of original pre-trained model Woo, and the second part is randomly initialized by Xavier method [Glorot and Bengio, 2010].

Learning

A VAE-based model will often ignore the latent variables, causing the posterior to collapse. For a text generation task, Zhao et al. [2017] utilized the bag-of-word (BOW) loss to alleviate this problem and thus achieved improved performance. In this paper, we introduce a new auxiliary loss that uses the posterior latent variable zt to predict the corresponding word in every generation step to preserve information.

Our final loss function is a weighted sum of Laux t and Lelbo t at each time step t:

where α is the weight controlling auxiliary loss, LLM t is the log-likelihood loss when predicting Yt, and LKL t is KL divergence of the approximate posterior distribution qt and prior distribution pt-1. The two losses are calculated as follows:

In the generation phase, we predict the token Yt by computing p (Yt|Y<t, z′t-1 ,C, e), in which the posterior latent variable zt is replaced with the prior latent variable z′t-1 .

4 Experiment

Since there is no off-the-shelf multi-turn emoji-rich dialogue dataset available, we collect a large corpus of Twitter conversations with emojis. We use 58 common emojis as the labels, consistent with the selection used in Mojitalk [Zhou and Wang, 2018] except that the six emojis with the lowest frequencies are removed. Each conversation in our corpus consists of at least three rounds, where the first round belongs to the context and the last round is Ue containing an emoji. If there is more than one emoji in Ue, we select the most frequently used emoji as the emotional reaction label. When the frequencies are equal, we select the emoji with the lowest frequency in the corpus because it is most distinctive.

4.1 Data Preprocessing

During preprocessing, all mentions and hashtag tokens were removed, and repeated letters and symbols were shortened. To better capture the emotions of the conversations, we have added an external emoji vocabulary of size 1k, which is shared with the label space of the emotional reactions e.

The distribution of our emotional reaction labels e is close to that of Mojitalk (e.g., accounts for 34%). We have removed dialogues with fewer than 6 words and more than 34 words in the response Y and the emotional reaction utterance Ue. Dialogues with fewer than 6 words and more than 100

words in the context C have also been eliminated. We randomly split the corpus into 137,421/4,661/4,739 conversation pairs for train/validation/test set. For building validation and test dataset, we choose the response Y with emoji as label so as to test the emotional accuracy of the generated response. Note that the emotion labels of the response Y is not considered in train dataset, and all emojis in Y are all removed.

4.2 Implementation Details

We chose XLNet-base [Yang et al., 2019] as our pre-trained model; thus, dmodel is 768, and the input and output vocabulary sizes are 33k and 32k, respectively.

The Adam optimizer with an initial learning rate of 1e-5 was applied to all models. The batch size was set to 15, and greedy search was used for all methods. For the variational model, dz was set to 64, and the temperature used in the KL annealing strategy was varied from 0 to 1 in 10k steps. All experiments were conducted on a single 11 GB NVIDIA GeForce RTX 2080 Ti GPU card and took approximately seven hours at most to reach convergence.

4.3 Baselines

For all of the baseline methods, we use the same input vocabulary for comparison.

S2S: A simple seq2seq model based on GRU.

ECM: The emotional chatting machine which uses internal emotion memory and external emotion memory for emotion expression [Zhou et al., 2018].

Mojitalk: An encoder-decoder based CVAE model incorporated with emotion embedding [Zhou and Wang, 2018].

XLNet: A seq2seq model with an emotional reaction label e, and initialized with XLNet. Uni-directional attention is utilized for generation, as shown in Figure 2.

T-CVAE: A Transformer-based CVAE model. In contrast to the original T-CVAE implementation [Wang and Wan, 2019], our implementation is based on XLNet; therefore, the combination layer of T-CVAE is the same as that of EmoElicitor. We also add the BOW loss to alleviate model collapse.

4.4 Automatic Evaluation

Results for Response Generation

Embedding-based metrics: Y -avg [Liu et al., 2016] calculate the semantic similarity between the generated response and the ground truth response by averaging word embeddings based on Twitter word embeddings [Godin, 2019]. Ue-avg and Ue-Gre calculate the semantic similarity between the generated response and the ground truth reaction utterance Ue based on average and greedy matching, respectively.

Dist1 and Dist2: The proportions of distinct unigrams and bigrams [Li et al., 2016] in the generated responses.

BLEU and B1/2: Word-overlap scores against goldstandard responses. BLEU [Papineni et al., 2002] refers to BLEU-4, and B1 and B2 refer to other n-gram scores.

Acc5: We adopt the top-5 emoji accuracy [Zhou and Wang, 2018] to evaluate the agreement between the expected emoji category of the real response and the emotion category of the generated response predicted by a well-trained classifier. Note that the responses in the validation and test datasets have emoji labels that are conceptually different from the emotional reaction labels of Ue.

Avg-len: The average response length.

As shown in Table 1, our model performs best in terms of Y -avg, Ue-avg, Ue-Gre, Dist2, BLEU, Avg-len and Acc5. The models based on Transformer perform better than that base on RNN. In this table, 'w/o Ue' denotes the case in which Ue is eliminated and only the emotional reaction label e is used in the posterior net. The models that incorporate latent variables (i.e., T-CVAE and EmoElicitor) can generate more diverse responses. We observe that T-CVAE performs better than our model in terms of Dist1; this is because the longer responses of our model result in a larger denominator in the formula Dist1 = N dist1 /N total .

The models without Ue perform more poorly in terms of Y -avg, Ue-avg, Ue-Gre, BLEU, Acc5 and Avg-len, indicating that considering Ue can help to generate more coherent responses in conversation. However, the Dist metrics do not decrease as the others do. We conjecture that the posterior latent variables can benefit from Ue but that Ue has limited effects on the prior net, which contributes more to the diversity. On the other hand, the Dist metrics of our model without the pre-trained model do decrease. This finding confirms the effectiveness of the pre-trained model in learning the dependency on context.

Results for Emotion Elicitation

To evaluate model's emotion elicitation ability, for each selected 58 emojis we generate one response Ye in reply to each context C in test set.

Acc10 is the prediction accuracy for the emotion e that is elicited based on the generated response Ye and context C. Note that we employ Acc10 (top-10 emoji accuracy) because elicited emotion prediction is much more difficult than response emotion classification.

Pro@5 is the sum of the probabilities of the top 5 emotion labels for every context C. We built another classifier to predict the emotional reaction e based on a given context C.

In Table 2, the Acc10 score of our method increases without the pre-trained model. This is because our classifier can more easily judge emotional words (e.g., love you ), whereas the words generated by the pre-trained model are more diverse. Pro directly calculates the probability of eliciting emotions, which is fair across all categories, while Acc10 often ignores some difficult-to-judge categories (e.g., ). Our model scores highest in terms of Pro and Pro@5, indicating that it is the model that is most effective in eliciting all desired emotions.

4.5 Manual Evaluation

To better evaluate the quality of the generated responses, we performed manual evaluation. Given a post and an emotional reaction label, responses generated from all the Transformerbased models were randomized and presented to five graduate students majoring in sentiment analysis. We randomly sampled 200 posts from the test set and selected a specific emoji as the target emotional reaction label to generate a response in accordance with the emoji distribution.

Each annotator was asked to score each response in terms of each of the following metrics with a rating of 0 or 1: (i) Grammar (Gram): whether the generated response is natural and fluent. (ii) Topic coherency (TC): whether the response is topically coherent to the context and reasonable in logic. (iii) Emotional consistency (TC): whether the response can elicit the given emotion.

Results: We calculated the Fleiss' kappa [1971] to measure interrater consistency. Fleiss' kappa for Gram, TC and EC are 0.84, 0.60 and 0.58, indicating substantial, moderate and moderate agreement respectively. As shown in Table 3, the performance of our model is consistently in line with the human perspective. Notably, XLNet can elicit emotions with only 38.2% success because of the imbalance of the emotion categories (e.g., accounts for 34%).

4.6 Case Study

We sample some generated responses from all three models in Table 4. The XLNet model simply generates the same responses for different emoji reaction labels because it is difficult for a single emoji to represent the user's reaction, which causes XLNet to ignore the emojis. T-CVAE always generates the same words at the beginning of each sentence (e.g., They are). We speculate that the single latent variable z has difficulty affecting the decoder output at the beginning. By contrast, our model generates diverse responses for different emoji (e.g., shot on their asses , heart attack ), thus demonstrating that latent variable zt at time step t is able to capture dependency between words and emotional reactions.

5 Conclusion

In this paper, we investigate emotional reactions, including next-round utterances and the corresponding elicited emotion labels, in dyadic dialogue generation and propose a novel model called EmoElicitor to generate responses with emotional reaction awareness. By incorporating a latent variable zt in every time step, we can capture the dependency between words and emotional reactions, thus allowing our model to generate coherent, diverse responses with the intent of eliciting different emotional reactions.

Acknowledgments

The work was supported by the National Key R&D Program of China under grant 2018YFB1004700, National Natural Science Foundation of China (61872074, 61772122).