Progressive Open-Domain Response Generation with Multiple Controllable Attributes

1 Introduction

Developing human-like conversational agents is a longlasting goal of artificial intelligence. Recently, thanks to the availability of a plethora of conversation data on the Internet and the booming of deep learning technologies, researchers have been attracted to explore end-to-end data-driven approaches to building social chatbots [Huang et al., 2020].

Nowadays, sequence-to-sequence (Seq2Seq) models [Serban et al., 2016] have been adopted to generate conversations due to their scalability and promising capability in capturing language-independence to implicitly learn semantic and syntactic relations between message-response pairs and contextual dependencies. However, they usually tend to generate "safe responses", such as "I do not know" and "OK", because the vanilla Seq2Seq models are prone to only memorize highfrequency responses in the data [Serban et al., 2017; Xing et To ensure the relevance of a response to the message and fidelity of the response to the controlled attributes, PHED designs subtle losses under rigorous mathematical derivation. Specifically, we utilize Transformer because it facilitates the self-attention mechanism for many NLP applications [Vaswani et al., 2017]. To ensure the diversity of the generated responses with controllable attributes, we apply Conditional Variational AutoEncoder (CVAE) and separate the CVAE latent variables into two meaningful types of variables: a joint latent variable capturing semantic features shared among all data and specific latent variables, each of which controls the attribute at the corresponding stage. The learned CVAE latent variables are then coupled with the encoding information learned at previous stages to explicitly promote the effect of the specific attributes in generating responses. Here, we borrow the idea of story completion in [Wang and Wan, 2019] to utilize the proved effective architecture of Transformer-based CVAE (T-CVAE) to implement the coupling procedure. Different from T-CVAE, PHED does not share the parameters in the encoder and the decoder, but contains more CVAE latent variables, which are optimized by a newly derived evidence lower bound (ELBO) and controlled losses. We conduct extensive evaluations and demonstrate that PHED can generate more diverse responses.

The contribution of our work is threefold: (1) a first work to generate diverse responses with multiple controllable nesting attributes; (2) a unified framework to include only one aspect of controllable attributes at one stage, relying on a hierarchical structure that enjoys flexibility and extensibility with rigorous theoretical guarantee; (3) empirical evaluations clearly demonstrating the effectiveness of PHED.

2 Our Proposal

We present PHED with the theoretical results and its training procedure.

2.1 Preliminaries

Given a corpus, D = {(xi, ci, yi)}Ni=1 , where N is the number of message-response pairs, xi = xi1 xi2 . . . xi|xi| is a message with |xi| characters or words, ci denotes the associated attributes on the response of yi = yi1 yi2 . . . yi|yi|, the objective is to learn the conditional probability p(y|x, c) from the corpus. Here, the attribute c = l1, . . . , lK enforces the attribute li at the i-th stage from K pre-defined aspects, e.g., the emotion of happy or sad [Jiao et al., 2019], and the tone of Declarative, Interrogative, or Imperative [Ke et al., 2018]. After obtaining p(y|x, c), given a message x and a specific attribute c, we will generate response y accordingly.

We propose Progressively trained Hierarchical EncoderDecoder (PHED), shown in Fig. 2(a), to enforcing controlling only one aspect of attributes from the data at one stage. The basic structure of PHED resembles T-CVAE [Wang and Wan, 2019], but distinguishes the CVAE variables to zc ∈ Rdzc for capturing common semantic features and zi ∈ Rdzi for capturing individual features at the i-th stage, where dzc and dzi denote the size of zc and zi, respectively.

To relieve the burden of the model expression, we define the vanilla Transformer layer for the encoder (Te) and the decoder (Td) as follows:

Here, all the hidden size in the Transformer layer is H. hi , hid ∈ RH denotes the output of the encoder and the decoder at the i-th Transformer layer, respectively. MHEe and MHEd is the multi-head attention network with the input of query, key, and value in the encoder and the decoder, respectively. LN denotes the operation of layer normalization and FFN is a feed forward neural network.

Borrowing T-CVAE in [Wang and Wan, 2019], we define:

Hence, z is sampled from an isotropic Gaussian distribution with mean (µ) and variance (σ) computed in two steps. In Step I, a hidden feature v is computed from a multi-head attention network on Ψ, MHΨ, which takes three inputs, i.e., α (a random initialized context vector) for the query, and a for the key and the value, respectively. In Step II, v is fed to a multi-layer perceptron (MLP) to determine the mean and the variance simultaneously.

2.2 Model and Theory

Let hi ∈ RH and Edi dec ∈ RH be the output of the encoder and the decoder at the i-th stage, respectively. di is the number of layers in the decoder up to i-th stage, i = 1, . . . ,K. When i = 0, we compute h0x and h0y by the first Transformer layer:

where ˜x is the sum of token embedding and position embedding on x, i.e., WE(x) + PE(x).

The decoder of the first Transformer layer is computed by

The corresponding CVAE variables at the i-th stage of the recognition network and the prior network are:

Obviously, the difference between the two networks lies in whether the multi-head attention network attends to the decoder h0y or not. It is noted that h0x and h0y , rather than hi-1 , are applied to learn the parameters of both networks because hi-1 has absorbed the attribute information in all previous stages and may contaminate the original data.

We train PHED in the following progressive way:

1. The CVAE variables z1c and z1 are sampled from the recognition network defined in Eq. (6) by setting i = 1. Next, h1 and Ed1 dec are then computed by the newly stacked Transformer layer on the concatenation of h0 , z1c , and z1 (i.e., ˜h1 = [h0 ; z1c ; z1]):

We highlight two remarks: (1) The effect of the CVAE variables zc and z1 is realized by the multi-head selfattention on ˜h1 . (2) The input of Td is slightly different from the standard Transformer. That is, the selfattention in PHED is applied at the same stage, not from scratch. It can then enhance the impact of the CVAE variables at the corresponding stage.

1. At the i-th stage (i ≥ 2), we fix the parameters learned at previous stages and sample zic and zi from the recognition network defined in Eq. (6) and compute hi and Edi dec by a newly stacked Transformer layer:

where ˜hi = [hi-1 ; zi]. Note that ˜hi does not include zic because it has been absorbed in hi-1 .

1. Step 2 continues until we reach the K-th stage. The parameters are learned by the Multi-stage Training procedure detained in Sec. 2.3.

By the above generation mechanism, we can derive the fol-

lowing theorem to compute the conditional probability:

Theorem 1. Given the above defined notations, the conditional generation probability can be computed by

and the evidence lower bound at the i-th (i ≥ 1) stage is

The proof is provided in the Appendix. Eq. (10) holds due to the variable dependency and the Markov chain on hi . Here, we only consider the Markov property and leave the variants of including more hidden states as a future work. Note that in Eq. (11), the derived ELBO (LELBO) consists of not only the expected log-likelihood estimator, LiM , but also two separated KL divergences, Lc,i KL and LiKL to control zic and zc.

2.3 Losses and Training

Other than the derived KL divergence in Eq. (11), we need the following losses to constrain CVAE latent variables sampled from Eq. (6). First, zic and zi should be as dissimilar as possible. Moreover, to balance the effect of zc and zi, we force their length to be nearly the same and yield the following loss:

Second, we expect that zc changes little across two consecutive stages and enforce it by minimizing the Fr´echet Inception Distance (FID) [Heusel et al., 2017]:

This loss is also equivalent to minimizing the Wasserstein2 distance on two isotropic Gaussian distributions, i.e., the sum of the difference of mean and standard deviation on two Gaussian distributions.

Third, to guarantee encoding meaningful information, we follow the idea of [Zhao et al., 2017] to enforce the bag-ofword (BOW) loss on zic :

where ybow are the words in response y without order, and p(ybow|hi , zic ) is obtained by a single layer fully-connected network hb = MLPb([hi ; zic ]).

Fourth, the cross entropy loss is placed to guarantee the effect of the fed attribute:

Multi-stage Training

We train PHED progressively: at the first stage, we estimate the model parameters by minimizing the following loss:

where λ is gradually increased from 0 to 1 via the annealing technique [Ke et al., 2018] because L1M , Lbow z1c , and L1cls are cross entropy losses with nearly the same scale while the effect of Lz1 is small as observed.

Next, at the i-th stage (i≥2), we freeze previously learned parameters and seek new parameters by minimizing

where the loss Lzic is specially included to guarantee the smoothness of the change of zic . The above minimization procedure continues until i reaches K.

After training PHED, given a message x, we can then generate each type of responses with the associated controlled attribute at each stage. That is, we sample zic and zi from Eq. (7) and concatenate them with hi to construct the input of Transformer at each stage, i.e., ˜h1 = [h0 ; z1c ; z1] for Ed1 dec as in Eq. (8) and ˜hi = [hi-1 ; zi] for Edi dec as in Eq. (9), where i = 2, . . . ,K. Let Ek dec,t be the k-th stage decoder at the t-th step, we can generate the response by

where Wo ∈ RH×|V | is the parameter shared with the embedding layers and |V | is the vocabulary size.

3 Experiments

We conduct experiments to address the following questions: (1) What is the performance of PHED in both automatic and human evaluations? (2) What is the effect of the losses in PHED? (3) What are the generation results?

3.1 Data

The data is the short-text conversation dataset (STC) [Shang et al., 2015], collected from Sina Weibo, a Chinese social platform. After setting the maximum number of characters in a response to 30, we obtain around 3.9 million dialog pairs and split them into the set of training, validation, and test with the ratio of 90%, 5%, and 5%, respectively. We pick three independent aspects of attributes, Emotion (Emo.), Tone, and length (Len.). The emotion aspect consists of six categories: angry (A), disgust (D), happy (H), like (L), sad (S), and others (O). The tone aspect considers three types: declarative (D), interrogative (I), and imperative (M). The emotion classifier and the tone classifier is trained as in [Zhou et al., 2018; Ke et al., 2018]. Based on the typical length generated by the baselines, we set the length of a response as long, denoted by L, when the number of characters is greater than 12 and others as short, denoted by S. Table 1 reports the data statistics.

3.2 Methods and Implementation

We compare our PHED with the following strong baselines: (1) MMI-bidi [Li et al., 2016]: a popular Seq2Seq LSTM model applying the maximum mutual information objective to re-rank responses from beam search; (2) SC-S2S [Zhang et al., 2018]: an Seq2Seq LSTM model with the specificity controlling; (3) DCVAE [Gao et al., 2019]: a newly proposed Seq2Seq LSTM model with CVAE to generate responses through two-stage sampling. (4) T2T [Vaswani et al., 2017]: a Transformer-based baseline, which is also the base of PHED without controllable attributes.

Our implementation is in PyTorch 1 . We apply the default setup for the baselines with the best tuned parameters on the highest BLEU score. Following [Zhang et al., 2018], SC-S2S applies Jieba to construct a vocabulary dictionary in 40,000 most frequent Chinese words. The same as [Gao et al., 2019], DCVAE constructs a vocabulary dictionary with 50,000 most frequent Chinese words and characters. MMIbidi and T2T apply the same Chinese characters as the input to attain a dictionary with 9,500 Chinese characters out of total 10,549 characters extracted from the dataset, where the low-frequented 1,049 characters are denoted by <UNK>. Moreover, three additional special tokens, <EOS>, <BOS>, and <PAD> are introduced to denote the start and the end of a sentence, and the padding character, respectively. By this setting, we obtain a much smaller size of the dictionary in T2T and PHED, yielding a lighter effort to learn precise representations for the input tokens. For each Transformer block, we set the number of self-attention heads to 8 and the hidden size (H) to 512. T2T consists of 6 Transformer layers in both the encoder and the decoder, pre-trained on 40

1 https://www.dropbox.com/s/1376kmhvuaxqe5h/PHED.zip?dl=0

million message-response pairs. PHED applies CVAE on top of T2T. The size of CVAE latent variables is set to 128, i.e., dzc = dzi = 128. We stack two more Transformer layers for each aspect of attributes in the order of emotion, tone, and length and yield a total of 12 Transformer layers in PHED. The order of fed attributes is not optimized but follows the proved humans' effective learning procedure [Spiering and Ashby, 2008], from difficult tasks to easy ones. Except for the parameters in the initial Transformer block, the remaining parameters are initialized by the Xavier method and trained by ADAM with the learning rate 0.0001 and the batch size of 32. In the inference, we set the beam search size to 5. Under the above settings, we train PHED 10 epochs at each stage on a Tesla V100 GPU and cost about 51 hours.

3.3 EvaluationMetrics

We evaluate the models by: (1) BLEU: BLEU-n measures the average n-gram precision on a set of reference sentences. As DCVAE [Gao et al., 2019], we set n = 1, 2, 3, 4. (2) Dist. 1 & Dist. 2 [Li et al., 2016]: the ratios of distinct unigrams and bigrams in the generated responses to the total generated unigrams and bigrams, measuring the diversity of the responses. For a fair comparison, all metrics are evaluated by the Chinese-character-level tokenization. (3) Human evaluation: Three expert labelers were recruited to evaluate the generated responses for 300 randomly selected posts based on the following 4-point criteria: 1) +3: the response is not only semantically relevant and grammatically correct, but also informative and interesting; 2) +2: the response is grammatically correct and can be used as a response to the utterance, but is too general (e.g., "OK"); 3) +1: the response is grammatically correct, but semantically irrelevant; 4) +0: the response contains mistakes (e.g., grammatical errors or <UNK>). Though it is different from the 3-point criteria in [Zhang et al., 2018; Gao et al., 2019], the 4-point criteria allows us to further distinguish meaningful responses from general and irrelevant responses. The values of the Fleiss' Kappa [Fleiss and Cohen, 1973] are great than 0.3 in all cases, which indicate the inter-rater consistency among three labelers.

3.4 Experimental Results

Table 2 reports 14 cases of PHED by selecting two typical types for each aspect of attributes and shows that

• Relevance. PHED attains the best two BLEU scores, which imply the generation relevance. Moreover, long responses can get significantly higher BLEU scores (p < 0.01 in ttest) than the short responses and the LSTM-based methods. Even short responses can attain competitive BLEU scores compared to the baselines.
• Diversity. PHED attains relatively lower scores in Dist. 1 because usually PHED generates longer responses than baselines, yielding a larger number of unigrams. In terms of Dist. 2, PHED attains significantly higher scores than the baselines (e.g., 4.61 vs. 3.07, around 50% gain). This again shows that PHED generates more diverse responses.
• Length. According to the results in the last column, PHED tends to generate longer responses, with more powerful expression ability than the baselines. Even when setting to generate short responses, PHED can generate longer responses than SC-S2S and DCVAE and similar length for MMI-bidi and T2T. When setting to generate long responses, PHED can generate 3 to 5 more characters for each response, significantly longer than the baselines.

The human evaluation results in the eighth to tenth columns of Table 2 are consistent with the automatic evaluation: (1) PHED generates significantly more relevant responses, where the values of 42% vs. 24% and 71% vs. 56% indicate that PHED generates more good (scoring over 3-point) and acceptable (scoring over 2-point) responses. (2) DCVAE and T2T are competitive while MMI-bidi and SC-S2S attaining much lower scores than them. Overall, PHED generates more good responses than baselines.

3.5 Ablation Study

We conduct ablation studies on PHED. The test shows that PHED produces similar results on all the combinations of attributes. In Table 3, we only report the scores of BLEU4, Dist. 2, the accuracy of emotion (Emo.), tone (Tone), and the length prediction (Len.A.), and the average length (Len.) in five models: T2T, PHEDS, PHEDSI, PHEDSIS and PHEDSIL; see more results in the Appendix.

The results show that: (1) The Dist. 2 scores in PHED on different fed attributes are all higher than that in T2T. This means that PHED attains more diverse responses than T2T. By examining more details, we can observe that the emotion accuracy increases slightly after adding more other attributes. The tone accuracy is around 90% while the length accuracy is at least 84%. (2) By removing the losses related to zc in PHED, we obtain similar BLEU-4 and Dist. 2 scores to PHED with all restricted losses but attain slightly higher accuracy on all three aspects of attributes. The results imply that by removing zc, we can promote the effect of individual attributes and yield better attribute accuracy. (3) By examining the results of only removing classification loss (i.e., -Licls ), we observe that the corresponding attribute's accuracy slashes largely and becomes normal when removing both Lzi and Licls . The observation means that Licls plays an essential role in controlling the attributes when PHED needs to satisfy other minimization requirements.

3.6 Case Study

Figure 3 illustrates a complete examination on the compared methods in Table 2. Our PHED clearly generates responses with the specific attributes progressively, including not only the corresponding emotion aspect, but also the exact tone and the length. For example, in the Happy emotion, PHED frequently generates "Congratulations" and "Haha". While in the Interrogative tone, it generates the related words, e.g., "What" or "Why". Moreover, by changing the required response length from short to long, more characters can be produced. For example, in Post#1, PHEDSDL generates similar words in the beginning to PHEDSDS but produces one more sentence to enrich the expression than PHEDSDS . By examining all responses generated by PHED, the fidelity to the attribute(s) is clearly confirmed. In terms of the responses generated by MMI-bidi, SC-S2S, and DCVAE, they are usually shorter and blank. Responses generated by T2T are a little fluctuation and cannot deliver any attribute effect. More examples can be shown in the Appendix.

4 Conclusion

We propose Progressively trained Hierarchical EncoderDecoder to generate responses with multiple controllable attributes. By incorporating CVAE into Transformer, we represent the controlled attributes by a joint latent variable and further specific latent variables, where the CVAE latent variables are then coupled with the encoding information to generate responses. The model is then effectively trained progressively by maximizing the evidence lower bound while minimizing several subtly designed losses. Empirical results with both automatic and human evaluations demonstrate that PHED significantly outperforms the state-of-the-art neural generation models and is able to generate more diverse responses.

Several challenging but interesting directions will be considered in the future. First, we only exploit three aspects of attributes in one order of generation. It is practicable and useful to further take into account other aspects and other orders. Second, we only apply CVAE under the Markov assumption. It is interesting to explore more dependencies in CVAE. Third, the current task only focuses on open-domain response generation. It would be worthwhile to probe other tasks, e.g., text generation in the same spirit.