Adaptively Multi-Objective Adversarial Training for Dialogue Generation

1 Introduction

The end-to-end neural systems [Serban et al., 2016; Luan et al., 2016] based on SEQ2SEQ framework [Sutskever et al., 2014] and using the maximum likelihood estimation (MLE) objective have been extensively studied because of its ability to generate unseen conversations and high scalability. But the dialogue responses generated by these models tend to be generic, dull, repetitive [Tao et al., 2018; Li et al., 2016; Zhang et al., 2018] in practice. Recently, sequence generative adversarial nets (SeqGAN) [Yu et al., 2017] apply the promising adversarial training method of GANs [Goodfellow et al., 2014] on sequence generation task by employing policy gradient and Monte Carlo (MC) search. AdverREGS [Li et al., 2017] employs adversarial training to actual dialogue generation task and trains the generator against a real-fake discriminator. DP-GAN [Xu et al., 2018] and DAL [Cui et al., 2019] try to increase the diversity of the generated responses by designing a fine-grained diversity discriminator and utilizing the duality between query and response generation, respectively.

∗Equal contribution.

†Corresponding author: Rui Yan (ruiyan@pku.edu.cn)

But in fact, assessing dialogue responses should consider many aspects of linguistics such as diversity, fluency, syntax habits of human, and so on. However, although a welldesigned discriminator can make the generator perform well in the corresponding expected aspect, other aspects are often ignored. We notice that the multi-adversarial framework, denoted MDGAN, that extends GANs to multiple discriminators has achieved promising results on image generation tasks. Many models in MDGAN [Durugkar et al., 2017; Neyshabur et al., 2017; Albuquerque et al., 2019] use their specially designed methods to weight the sum of the losses of image discriminators to optimize the image generator overall.

We hope the generator can cover multiple linguistic aspects to achieve overall improvement and perform relatively well in all aspects. Intuitively, we can view dialogue generation as a multi-objective optimization problem [Deb, 2001; Albuquerque et al., 2019]. Each objective is designed for a linguistic aspect, and a corresponding discriminator provide guidances towards this objective, thereby optimizing the generator towards multiple objectives. However, this idea faces challenges. The main challenge is to innovatively propose a theoretically proven and practically effective framework that extends the original adversarial dialogue generator optimization problem to a multi-objective optimization problem, where each objective is defined by policy gradient. The second is to explore how to design and combine various linguistic objectives, and to deal with the problem of asynchronous optimization of different objectives during training.

In this paper, we propose the adaptively multi-objective adversarial dialogue generation framework using policy gradient, called AMPGAN, to push the generator Gθ to achieve overall improvement. AMPGAN models dialogue generation as a stochastic policy in reinforcement learning (RL) and directly performs gradient policy update to generate discrete dialogue utterances, and is formalized as the multiobjective optimization problem by framing the simultaneous maximization of rewards received from multiple discriminators. We design multiple objectives of linguistics for guiding Gθ of AMPGAN: indistinguishable from the human utterances, high syntactic score and diversity. We propose multiple corresponding discriminators and an adaptively adjusted sampling distribution to dynamically organize them to participate in adversarial training. Through the adaptive distribution, AMPGAN can focus on the aspects that Gθ not perform well relatively. When Gθ is still weak and have much room to improve in aspect ai, AMPGAN will adaptively increase the probability that the corresponding Di is selected to participate in adversarial training, to motivate Gθ to optimize towards ai; Conversely, when Gθ can easily deceive the discriminator Dj of aspect aj , AMPGAN will adaptively reduce its probability, and pay more attention to other relatively weak aspects. Thereby, Gθ can cover multiple aspects and solve the asynchronous optimization problem.

Our contributions are summarized as: (1) We reframe the dialogue generation as a multi-objective optimization problem and propose the novel multiple adversarial generation framework (AMPGAN) to improve the generator towards multiple objectives of linguistics and provide the theoretical derivation and proof; (2) We explore how to choose and combine different discriminators, and design an adaptively adjusting sampling distribution to balance all discriminators; (3) We conduct ample experiments on two datasets, and experimental results show the effectiveness of AMPGAN framework.

2 Methodology

We propose the adaptively multi-adversarial dialogue generation framework using policy gradient, called AMPGAN, to push the dialogue generator to improve towards multiple objectives based on linguistic apects, as shown in Figure 1.

2.1 The AMPGAN Framework

Given a dialogue query utterance x = {xi}mi=1 of m words, the generator Gθ needs to produce a response y = {yj}nj=1 of n words, where xi, yj ∈ T (the word vocabulary). AMPGAN conducts gradient policy update directly by modeling Gθ defined by a SEQ2SEQ model as a stochastic policy in reinforcement learning (RL). Each discriminator in AMPGAN is designed to judge on the complete generated sequence according to a linguistic objective, and reports the RL reward which is passed back to the intermediate state-action steps using MC search. Note that all discriminators {Di}Ni=1 do not share parameters. We extend and divide the types of discriminators into three categories, binary classification task DC, regression task DR and direct rule scorer DI . Among them, DC and DR are parameterized models that need training, and DI is a set of scoring logical rules which can be understood as a fully trained and completely correct discriminator.

More formally, Gθ and {Di}Ni=1 play the following minimax optimization game:

where the random variable discriminator D obeys the distribution π(Q) on the increment Q of RL rewards. x is the ground truth data and z is the query data drawn from a prior pz(z). Gθ maps z to the fake responses which obeys the distribution pg. Next we will provide formal theoretical analysis.

Theoretical Analysis

Firstly, we consider the optimization of multiple discirminators for any given Gθ and π(Q) under the proposed AMPGAN framework. Note that since the rule scorer DI is constant, here we only need to consider the variable DC and DR.

Proposition 1. For Gθ and π(Q) fixed, the optimal discriminator Di is: Di ∗ = pd(x)/(pd(x) + pg(x)).

Proof. According to the induced measure theorem [Somesh Das Gupta, 2000], the two expectations are equal: Ez∼Pz [f(G(z))] = Ex∼PG [f(x)]. The objective function can be rewritten as below:

Let A := ∑Ni=1 pDi (pd(x) logDi(x) + pg(x) log(1 -Di(x))) then diag({ pDi (pd(x)(Di-1)2+pg(x)Di 2 ) Di 2(Di-1)2 }Ni=1) is the Hessian matrix of -A, and all leading principle minors of the matrix are greater than 0, thereby -A is a convex function. So the maximum point D∗ i of A is stationary points on [0, 1]:

And D∗ i is also the optimal point of V (G, {Di}Ni=1) Q.E.D.

Secondly, we fix Di = D∗ i and find the optimal solution G∗ for the generator of the proposed AMPGAN framework.

min C(G) = - log 4.

Proof. Bring D∗ i into the above formula:

when pd and pg are consistent, get the minimum value - log(4) Q.E.D.

Thirdly, we directly use a policy gradient of RL to optimize Gθ, which can avoid the problem of differentiation difficulty [Yu et al., 2017] for discrete dialogue data in the traditional GAN naturally. Each Di in AMPGAN guides to improve Gθ by computing a specific reward signal on sequences generated by Gθ using MC rollout strategy. Specifically, starting from the current state s = Y1:t-1, action a = yt, using MC search until the complete sequence is generated, the search sequence set MC Gθ (Y1:t; N) is obtained. And the value function QGθ Di of single discirminator Di on this set is set to the average discrimination score as:

where Y1:T ∈ MC Gθ (Y1:t; N), M is the simulations number of MC. When there is no intermediate reward, we can use the average value JDi (θ) assigned by the expectation of value function QGθ Di to evaluate the policy Gθ [Sutton et al., 2000].

In AMPGAN framework, dialog generation is viewed as a multi-objective optimization problem. Formally, the generator solves the following multi-objective problem:

Specifically, we design an adaptively adjusted distribution D ∼ π(Q) to organize the multiple objectives as:

So that AMPGAN can continue to pay attention to the objectives that Gθ not performs well relatively and Gθ can be balanced and improved towards many objectives more stably.

Formally, when Gθ is weak in an objective, AMPGAN dynamically strengthens the adversarial training between Gθ and the corresponding Di by adaptively adjusting the sampling distribution π(Q) = {p(k) (D = Di)}Ni=1 as:

where Z = ∑Ni=1 p(k-1) i exp(∆QGθ Di (k) ) and ∆QGθ Di (k) = QGθ Di (k) - QGθ Di (k-1) + N(0, σ) and N(0, σ) is Gaussian noise, and initial π(Q) is p(0) (D = Di) = 1N . When ∆QGθ Di is larger, that is, there is much room for Gθ to improve in the corresponding objective, AMPGAN will select Di with a larger probability. Then the gradient of (8) can be computed as:

We can use θ ← θ + αk∇θJ (θ) to update the parameters θ of the generator, where αk denotes the learning rate. And in practice, we estimate the expectation in the Eq. (10) by sampling discirminators according to π(Q).

Algorithm 1 Training AMPGAN.

Require: generator policy Gθ; N discriminators {Di}Ni=1 ;

sampling distribution π(Q); dataset S = {X1:T } 1: Randomly initialize Gθ, {Di}Ni=1 , {π(Di) = 1N }Ni=1 . 2: Pre-train Gθ using MLE on S 3: Generate negative samples using Gθ 4: Pre-train {Di}Ni=1 separately 5: while MPGAN do not converges do 6: Sample a discirminator Di using π(Q) 7: for g-steps do 8: Generate a sequence Y1:T = (y1, ..., yT ) ∼ Gθ 9: for t in 1 : T do 10: Compute Q(a = yt; s = Y1:t-1) by Eq. (5) 11: end for 12: Update generator via policy gradient Eq. (10) 13: end for 14: for d-steps do 15: Combine negative examples generated by current Gθ and positive examples sampled from S to train Di 16: end for 17: Update π(Q) by Eq. (9) 18: end while

In summary, Algorithm 1 shows the details of AMPGAN. Before adversarial training, we first pretrain the generator Gθ using MLE on training set S, and then use Gθ to generate negative examples. We sample positive examples from S and combine them with negative ones to pretrain parameterized discriminators {Di}Ni=1 . During adversarial training, Gθ and each discriminator are trained alternately. At each training step, a discriminator Di will be selected from {Di}Ni=1 to participate in adversarial training according to distribution π(Q). We sample positive examples from S and use Gθ to synthesize corresponding negative ones and use both to train Di. Gθ is updated by using the policy gradient and MC search based on the expected final value function received from Di. Finally, we adaptively adjust the sampling distribution π(Q) based on the increments of the generator's scores in various aspects. Following [Li et al., 2017], we employ the Teacher Forcing strategy to tell Gθ what sequences are good and guide Gθ push itself to produce these good sequences by exposing the good target sentence to it directly.

2.2 AMPGAN for Dialogue Generation

We train the dialogue generator Gθ against 3 discriminators.

Generator

We use a standard SEQ2SEQ [Sutskever et al., 2014] model implemented by two-layer LSTMs with attention mechanism [Luong et al., 2015] to implement the generator's policy Gθ [Li et al., 2017]. Gθ encodes the input queries into hidden states, and then decodes them into output responses.

Real-Fake Discriminator

We continue the previous works [Li et al., 2017] and use the generic Real-Fake discriminator Drf to judge whether an input dialogue utterances {x, y} are generated by humans (denoted as a real label) or machines (fake). Drf is essentially a two-classifier, and uses a hierarchical encoder [Serban et al.,

2016] with a two-class softmax layer to report the probability Q+ that {x, y} are real and Q- that {x, y} are fake. The purpose of using Drf is to encourage Gθ to generate dialogue utterances that are difficult to distinguish between real or fake by taking Q+ as the reward.

Syntactic Discriminator

We notice that the generated sequences are often confusing because the arrangement of words is not in line with human syntax norms. We try to improve the syntax score of the generated sequences by feeding Gθ the syntactic reward reported by the proposed syntactic discriminator Dsyn. Following [Vashishth et al., 2019], We use graph convolutional networks (GCN) to extract syntactic features embedding esyn of a sequence. We use the Stanford CoreNLP parser [Manning et al., 2014] to parse each word sequence s = {x1, x2, ..., x|s|} into a dependency parse graph G = (Vs, Es), whereas Vs = {x1, x2, ..., x|s|} and Es stands for the labeled directed dependency edges (xi, xj , lij), where lij is the dependency relaiton label of xi to xj . We use the corresponding word embedding vi ∈ Rd to initialize each node embedding h0i ∈ Rd in Vs.

In graph convolutional layers of Dsyn, we employ the SynGCN [Vashishth et al., 2019] model with Edge Label Gating Mechanism to aggregate each node. Then embedding hki ∈ Rd of node i after k GCN layers is computed as hki = SynGCN(h0i ). We use max-pooling operation to obtain syntactic embedding esyn = {em}dm=1 , where em = max{hkim}|s| i=1 . Finally, syntactic score ssyn is computed as: ssyn = σ(BN(Wsesyn + bs)), where Ws ∈ R1×d is a parameter vector, bs is a bias, BN is a batch normalization layer [Ioffe and Szegedy, 2015], and σ(·) is the sigmod function. We use ssyn as the reward to guide Gθ to produce utterances that conform to human expressions.

Information Discriminator

The generator tends to produce high frequency words. We design the information discirminator Dinfo to directly compute the normalized information amount sinfo of the generated utterance with n tokens. Formally, sinfo is computed as sinfo = 1n ∑ni=1 I(wi)-I(wmin) I(wmax)-I(wmin) , whereas I(wi) =

-log p(wi), p(wi) is the frequency of word wi, wmax and wmin are the two words with the largest and smallest word frequencies in the corpus, respectively. We use sinfo as the reward to push Gθ to explore more novel words.

3 Experiments

3.1 Datasets

Cornell Movie Dataset∗ (denoted as S1) contains a large metadata-rich collection of fictional conversations extracted from raw movie scripts. It consists of 220579 exchanges between 10292 pairs of characters in the movie, involving 9035 characters in 617 movies, a total of 304713 utterances. OpenSubtitles Dataset (S2) is a well-known human-human scripted dialogue dataset. It is extracted frommovie subtitles which are not speaker-aligned [Tiedemann, 2009]. We use the English short message data, which contains about 140M utterances in total, covers 106K movie works.

∗Cornell Movie-Dialogs Corpus

3.2 Training Details

We sort the words in each corpus by word frequency and use the first 35K words with high word frequency as the vocabulary. We filter out dialogue utterances with less than 5 or more than 40 tokens. We set the training batch size to 128, the size of word embeddings and the graph node embeddings to 512, the hidden size of hidden layers of all encoders in all models to 256, and the number of all LSTM layers to 2, and GCN layer number of Dsyn to 1. We train all models using Adam optimizer [Kingma and Ba, 2015], and all the dropout rates to 0.7. For the generator Gθ, we use the learning rate decay strategy and set the initial learning rate to 0.1 and the decay factor to 0.99. We set the learning rate to 0.001 fixedly for Drf and Dsyn. In pre-training, we first pre-train Gθ 5000 iterations, then use Gθ to produce 2500×128 negative examples and sample the same amount of positive ones from the dataset correspondingly. We combine both to pre-train Drf and Dsyn. In MC search, we employ the experience [Li et al., 2017] that given a partially decoded sP , Gθ will keep sampling tokens in word distribution until decoding is complete. Repeat this process k times (k is set to 7) and obtain k sequences sharing a common prefix sP . The average of the corresponding k scores given by the discriminator is used as the reward. During training of each model, if the performance on the validation set has not improved for a long time, stop training and choose the checkpoint with the best performance.

3.3 Experiments Results

We use the following models as the baselines. (1) MLE trains a standard SEQ2SEQ generator Gθ using the traditional MLE; (2) PG-BLEU [Yu et al., 2017] trains Gθ by policy gradient using the BLEU score of generated sequences as the reward directly, thereby obtaining a higher BLEU score; (3) AdverREGS [Li et al., 2017] is a standard adversarial model that Gθ is trained against a real-fake discirminator alone, and can be viewed as an extension of SeqGAN on the SEQ2SEQ framework; (4) DP-GAN [Xu et al., 2018] designs a finegrained diversity discriminator to guide Gθ to produce sequences with high diversity. (5) DAL [Cui et al., 2019] tries to increase the response diversity by utilizing the duality between query and response generation.

We regularly combine different discriminators to guide the generator: (1) D1: only use single Drf , noted as AdverREGS; (2) D2: only use single Dsyn; (3) D3: only use single DInfo; (4) D1&D2: combine Drf and Dsyn; (5) D1&D3: combine Drf and DInfo; (6) D2&D3: combine Dsyn and DInfo; (7) D1&D2&D3: combine Drf , Dsyn and DInfo.

We use two categories of metrics to evaluate each model, namely Adversarial Evaluation [Li et al., 2017; Bowman et al., 2016] and widely used Common Evaluation metrics.

Adversarial Evaluation

Adversarial evaluation helps to analyze the impact of different discriminator combinations of AMPGAN. Following [Li et al., 2017], we train automated machine evaluators to distinguish machine-generated responses from human-generated responses, such as accurately distinguishing real or fake, giving high scores to real responses and low scores to the fake. We design three machine evaluators, namely RealE, SyntaxE and InfoE, corresponding to Drf , Dsyn and Dinfo, to report the ratio FoolRate of the cases that RealE are deceived, the syntax score SynScore, and the ratio InfoRate of the information amount of the generated responses to the ground truth, respectively. Note that each evaluator is in the same architecture as its corresponding discriminator, but trained separately using supervised methods and do not participate in adversarial training. The abilities of parameterized RealE and SyntaxE are 0.09 and 0.11, indicates how much we can trust the evaluation results given by the evaluators, and the smaller the value, the better. We also report the comprehensive score NormAve of SynScore, InfoRate and FoolRate as 13 ∑3i=1 (xij - ximin )/(ximax - ximin ), where xij , ximax , and ximin are the score of model j, maximum and minimum in the table on the i-th indicator respectively.

Table 1 shows the evaluation results. From Table 1 we can find out that AMPGAN can effectively improve the generator Gθ. Specifically, it can be summarized as follows: (1) Using a single discriminator is risky. We can find each model can fool the real-fake evaluator RealE to a large extent (greater than 97%) on S1, which shows that generators can easily perform well in this objective on S1, even without the guidance reward of Drf . (2) The generators against a single discriminator can go deeper towards expected objectives. We can find the generator against single D1 or D2 or D3 performs better in its corresponding objective than other single adversarial training models. (3) The generator trained against multiple discriminators can find a balance and achieve an overall improvement towards multiple objectives, although it may be weakened unilaterally. We can find the generator against D1&D2&D3 achieves significantly higher overall performance, although unilateral indicator values decrease a bit. Empirically then, we recommend that designers should design a general objective with the corresponding discirminator to report the basic but universal reward and supplement it with other specific rewards according to actual needs, so that the model can perform well in general and also stand out in several areas.

Common Evaluation

The metrics of common evaluation are as follows:

(1) BLEU score [Papineni et al., 2002] is widely used in sequence generation tasks [Li et al., 2016], measuring the overlaping between the generated word sequences and the ground truth; (2) Distinctness metric proposed by [Li et al., 2016] measures the diversity of generated token sequences. (3) Human Evaluation (HM): We ask three human annotators with a linguistic background to report on the overall quality of the responses to 200 examples randomly sampled from the test set, which are generated by the full AMPGAN model and all baseline models. (4) Similarity&Relevance: We also report the similarity between the ground truth responses and the generated, and the relevance between the queries and the generated responses at semantic level by calculating cosine similarity of the two token sequence embeddings computed by the pretrained BERT [Devlin et al., 2018] model.

Table 2 shows the evaluation results. From Table 2, we can find out that AMPGAN framework do help achieve better results on these common metrics. Specifically, the generator against three discriminators performs very well. It achieves the highest BLEU-1 score and Dist-1, Dist-2, and Dist-3 scores, as well as relatively higher BLEU-2 and Dist4 scores on both datasets, indicating that the generated token sequences have higher overlap with ground truth responses and higher diversity. It also achieve the highest ranking of Human Evaluation over baselines on both datasets. Moreover, the pairwise adversarial training generators also achieved higher word overlap and diversity than the corresponding single adversarial training ones separately.

Figure 2 shows the results of similarity&relevance evaluation. We can clearly find the combination of D1&D2&D3 makes the generated sentences obtain the highest similarity with ground truth responses and the highest relevance with queries on both datasets, indicating the token sequences generated by this generator are closest to the ground truth at the semantic level. Moreover, we can also find out the trend that the generator against multiple combined discriminators can achieve higher similarity and relevance than the one against each of these discriminators alone.

Effectiveness of Adaptive Sampling Distribution

We also investigate the adjustment of the discriminator sample distribution during training, as shown in Figure 3, and conduct a comparative experiment between using adaptive distribution and fixed uniform distribution, as shown in Table 3. From Figure 3, we can find out that different datasets have different characteristics, and AMPGAN can capture these characteristics by adaptively adjusting the sampling distribution of all objectives. And during training, improvement degrees of Gθ towards different objectives are constantly changing, so it is very useful to dynamically adjust the distribution. In addition, from Table 3, we can find that compared with using a fixed uniform distribution, using the adaptive adjustment distribution helps AMPGAN achieve higher scores of adversarial evaluation and diversity on two datasets, which also shows the effectiveness of the adaptive distribution. Taking the OpenSubtitles curve as an example, when Gθ has been promoted sufficiently towards one objective and its improvement room becomes relatively small, then AMPGAN will reduce its probability and focus on other objectives, so as to achieve the purpose of balancing all objectives and improving overall performance. And because Dinfo is absolutely accurate and can basically not be deceived by Gθ, Dinfo can always provide reward to guide Gθ to improve, which is reflected in its high probabilities. However, the Drf has the risk of saturation [Xu et al., 2018]. From Cornell curve, we can find that there is no improvement room towards the objective of Drf soon, indicating that Gθ can quickly fool Drf .

4 RelatedWorks

SeqGAN [Yu et al., 2017] successfully employs the promising GAN framework into the discrete sequence generation task by utilizing policy gradient and MC search. AdverREGS [Li et al., 2017] employs adversarial training to guide the generator to produce dialogue utterances that difficult for evaluators to distinguish between human-generated and machinegenerated. Many models focus on increasing the diversity of the generated responses: DP-GAN [Xu et al., 2018] designs a fine-grained discriminator which has a better diversity evaluation method; The Adversarial Information Maximization (AIM) [Zhang et al., 2018] optimizes a variational lower bound on pairwise mutual information between query and response; And DAL [Cui et al., 2019] utilizes the duality between query generation and response generation.

Recent literatures have demonstrated promising results in image generation tasks for extending the basic GAN framework using multiple discriminators to train against the generator. D2GAN [Nguyen et al., 2017] combines KL and inverse KL divergence into a unified objective function and uses the complementary statistical characteristics of these divergences to effectively disperse the estimated density to capture multiple modes. The Generative Multi-Adversarial Nets (GMAN) [Durugkar et al., 2017] trains the generator against a softmax weighted arithmetic average of K different discriminators in the same model architecture. HVM [Albuquerque et al., 2019] proposes hypervolume maximization to efficiently optimize a weighted loss from all discriminators for the case of large neural networks.

5 Conclusions

In this paper, we propose the adaptively multi-objective adversarial dialogue generation framework using policy gradient, called AMPGAN, to push the generator to improve itself towards multiple objectives of linguistics by feeding these reward signals from the discriminators corresponding to these objectives. Comparative experimental results of different combination models and baseline models show that AMPGAN can significantly improve overall performance with only minor sacrifices in each objective.

Acknowledgments

This work was supported by the National Science Foundation of China (NSFC No. 61876196).