One "Ruler" for All Languages: Multi-Lingual Dialogue Evaluation with Adversarial Multi-Task Learning

1 Introduction

The open-domain dialogue system is of growing interest in the field of Natural Language Processing (NLP). Its central goal is communicating with humans coherently and meaningfully; it also has wide industrial applications like XiaoIce 1 fromMicrosoft. Significant efforts have been made in recent years, to develop large-scale non-task-oriented dialogue system [Serban et al., 2016; Li et al., 2016; Tian et al., 2017; Yao et al., 2017; Song et al., 2018]. These models adopt endto-end neural network systems to predict the next dialogue utterance by the maximum likelihood estimation (MLE), given the previous dialogue turns.

Meanwhile, previous research has developed some successful automatic evaluation metrics. For example, BLEU [Papineni et al., 2002] and METEOR [Banerjee and Lavie, 2005] are proposed for machine translation. ROUGE [Lin, 2004] is widely used in automatic summarization. However, when it comes to open-domain dialogue evaluation, these metrics are shown to correlate poorly with human judgments [Liu et al., 2016a]. Researchers have to use those wordoverlap metrics as there are few alternative efficient metrics

∗Corresponding author: Rui Yan (ruiyan@pku.edu.cn)

1 http://www.msxiaoice.com/

[Li et al., 2016; Yan et al., 2016]. Some researchers rely on the manual annotation to evaluate their models, but it is costly and time-consuming. Therefore, having an accurate automatic dialogue evaluation model is in great need.

Very recently, some efforts have been made to develop a neural network-based metric for dialogue evaluation [Lowe et al., 2017]. It learns to predict a score of a reply given its query (previous user-issued utterance) and a groundtruth reply. This method requires massive manual annotation. RUBER [Tao et al., 2018] tries to address the cost of annotation through negative sampling and incorporating with referenced method.

However, the above methods only extract features from monolingual corpus, in which the trained metrics are not flexible enough to transfer across different language evaluation tasks simultaneously. Besides, these methods do not exploit a multi-lingual representation to enrich the features for automatic dialogue evaluation.

In this paper, we propose an adversarial multi-task learning for multi-lingual dialogue evaluation by integrating shared knowledge from multi-lingual corpora. Specifically, we regard each monolingual evaluation as a single task and propose a shared-private model under the framework of multi-task learning [Caruana, 1998; Ben-David et al., 2003]. The multitask learning structure contains two kinds of spaces: private and shared. The private feature spaces are used to extract the language-specific properties while the shared feature spaces capture the language-invariant properties across languages. Besides, motivated by the success of adversarial learning in domain adaption [Ganin et al., 2016; Bousmalis et al., 2016; Chen et al., 2017], we incorporate adversarial strategy with shared spaces to enhance their ability to extract the common underlying features, and avoid the shared feature spaces being contaminated by noise.

The contributions of this paper could be summarized as follows:

• Multi-task learning is first introduced for automatic dialogue evaluation. It extracts not only language-specific features in private spaces but also language-invariant features in shared spaces across languages.
• An adversarial strategy is used to strengthen the ability to extract language-invariant features in shared spaces, in which a new objective function for multi-lingual dialogue evaluation is also proposed.

We evaluated adversarial multi-task neural metric (ADVMT) on both English and Chinese evaluation tasks. Experiments show that our proposed metric significantly outperforms existing automatic metrics in terms of the Pearson and Spearman correlation with human judgements, and has a boosted performance with the help of each monolingual evaluation task.

2 RelatedWork

2.1 Automatic EvaluationMetrics

From the machine learning perspective, automatic evaluation metrics can be divided into non-learnable and learnable approaches. Non-learnable metrics typically measure the quality of generated sentences by heuristics (manually defined equations), such as BLEU, ROUGE and Greedy Matching [Rus and Lintean, 2012]. As the valid reply in dialogue systems are of high diversity under a given context, these metrics are shown to correlate poorly with human judgments [Liu et al., 2016a] for dialogue systems.

Compared with non-learnable metrics, learnable metrics can integrate linguistic features to enhance the correlation with human judgments through supervised learning. Lowe et al. [2017] develops a neural network-based metric for dialogue evaluation. RUBER [Tao et al., 2018] addresses the cost of annotation through negative sampling and incorporating with referenced method. However, these metrics are trained in monolingual corpus, which are not flexible enough to transfer across different languages. Different from the above methods, our proposed metric extracts features from multi-lingual corpus and could be applied to different language evaluation tasks simultaneously.

2.2 Multi-task Learning with Neural Networks

The main concept of multi-task learning [Caruana, 1998] is to extract the common underlying features between related tasks and to improve the performance of each task with the help of private features and shared knowledge through parallel training. In recent years, researchers have incorporated it with recurrent neural networks (RNN) to address various NLP problems [Collobert and Weston, 2008; Hashimoto et al., 2017].

Liu et al. [2016b] proposes a generic multi-task framework, in which different tasks can share information by an external memory and communicate by a reading/writing mechanism. Inspired by the success of multi-task learning, we regard each monolingual evaluation as a single task and propose a shared-private model under the framework of multitask learning for multi-lingual dialogue evaluation.

2.3 Adversarial Neural Networks

Adversarial neural network [Goodfellow et al., 2014] includes a neural generator G and a discriminator D, which is trained to classify real data versus generated data. Recently, the idea of adversarial networks is applied to various NLP tasks.

Chen et al. [2016] applies adversarial deep averaging network to transfer sentiment knowledge learned from labeled English data to low-resource languages where only unlabeled data exists. Chen et al. [2017] proposes an adversarial multicriteria learning for Chinese word segmentation by integrating shared knowledge from multiple segmentation criteria. Liu et al. [2017] introduces an adversarial multi-task learning framework, alleviating the shared and private latent feature spaces from interfering with each other. Motivated by the success of adversarial networks, under the framework of multi-task learning, we incorporate adversarial strategy with shared spaces to enhance their ability to extract languageinvariant features and propose a new objective function for multi-lingual dialogue evaluation.

3 Methodology

Given a previous query q and a reply r, the goal of neural network-based metric is to automatically measure the relatedness between q and r with a predicted socre(q, r). In subsection 3.1 we introduce the neural network-based metric for monolingual dialogue evaluation, and regard it as a single task in our proposed multi-task learning framework in Subsection 3.2. In Subsection 3.3, we incorporate adversarial strategy to multi-task learning and introduce a new objective function for multi-lingual dialogue evaluation.

3.1 Neural Network-basedMetric for Monolingual Dialogue Evaluation

This subsection mainly considers a single task neural metric to predict the appropriateness of a reply with respect to a query, of which the main structure (Figure 1) is inspired by Tao et al. [2018]. As for each word in a query q and a reply r, we first map them into vector representations (embedding). Then bi-directional Long Short-term Memory [Hochreiter and Schmidhuber, 1997] (Bi-LSTM) unites with forward and backward directions are applied to capture information along the word sequence. The update of each Bi-LSTM unit can be written precisely as follows:

where xt and T denote the embedding of the current input word and the last time step, and -→h t is the forward hidden states. Likewise, ←- h t is the backward hidden states. ⊕ denotes the concatenation operation and all parameters in Bi-LSTM model is referred as θ. Specifically, we regard the concatenated output of both directions of Bi-LSTM, at the last time step T , as the representation of the whole sequence (q and r, respectively):

after which we concatenate q and r to match the two utterances. In addition, we include the "quadratic feature" that proposed in Tao et al. [2018], denoted as qT Mr, whereM is a parameter matrix. Finally we use a multi-layer perceptron (MLP) to predict a scalar score of the given conversation pairs. The MLP we adopted has two layers. The tanh is used as the activation function in the hidden layers of MLP, while the second layer uses sigmoid to make the score bounded.

In the process of training, we consider negative sampling to avoid costly manual annotation. Negative sampling is adopted for utterance matching in previous work [Yan et al., 2016; 2017] and is shown to be feasible, which could ease the burden of costly manual annotation. Concretely, given a groundtruth query-reply pair, we randomly choose another reply r- from the training set as a negative sample. The main goal of negative sampling training is to make the score of positive samples be larger than the negative samples by at least a margin δ. Thus the training objective is to minimize

3.2 Multi-task Learning for Multi-lingual Dialogue Evaluation

The method we introduced above only extract features from monolingual corpus, in which the trained metric is not flexible enough to transfer across different language evaluation tasks simultaneously and does not exploit a multi-lingual representation to enrich the features for automatic dialogue evaluation.

Inspired by the success of multi-task learning, we regard multi-lingual dialogue evaluation as multiple "related" tasks and propose a shared-private model, which shares information across languages. This shared-private mechanism is supposed to improve the performance of each other simultaneously with the help of shared features [Chen et al., 2017; Liu et al., 2017].

To enable multi-task learning for multi-lingual dialogue evaluation, as depicted in Figure 2, we design two feature spaces for both tasks A and B: a private space to capture language-dependent features, and a shared space to capture language-invariant features. Each monolingual evaluation task is assigned a private Bi-LSTM layer and a shared BiLSTM layer. Sentences are encoded by these two kinds of Bi-LSTM layers simultaneously.

Formally, for task k, the query vector representations of shared layer q(s) T and private layer q(p) T are formed as follows:

the reply vector representation of shared layer and private layer are denoted as r(s) T and r(p) T , likewise. As all the tasks share the shared layer, the formula of shared layer is indicated by subscript.

To compute the similarity of query-reply pair, in each monolingual evaluation task, the sentence representations from private layer and shared layer are concatenated as the final embedding. Specifically, for task k, the final sentence representations of query and reply are:

which are then concatenated to calculate the score(q(k), r(k)) for each monolingual evaluation task k.

3.3 Incorporating Adversarial Strategy for Shared Spaces

Although the shared-private model separates the feature space into shared and private spaces, there is no guarantee that sharable features do not exist in private feature space, or vice versa [Liu et al., 2017]. We hope that the features extracted by shared spaces is invariant across languages, under the multi-task learning framework for multi-lingual dialogue evaluation.

Inspired by the work on domain adaption [Ganin et al., 2016; Bousmalis et al., 2016], we exploit adversarial training strategy to optimize the shared layer, as shown in Figure 3. We use a discriminator to recognize which monolingual evaluation task the encoded sentence comes from. This discriminator maps the shared representation of sentences to a probability distribution, then makes a prediction of classes of monolingual evaluation tasks by its probability. The shared layers are designed to work defiantly towards a learnable multi-layer perceptron, preventing it from making an accurate prediction about the types of tasks. In this way, shared spaces are trained to be purer and less vulnerable to the contamination from private spaces.

Formally, for each monolingual evaluation task k, assume that there are Nk query-reply pairs. We refer to s(q) k,i and s(r) k,i as shared features from query and reply respectively, for i-th query-reply pair of task k.

We further concatenate s(q) k,i and s(r) k,i as the input of the discriminator, denoted as sk,i = s(q) k,i ⊕ s(r) k,i . Finally, the discriminator computes the probability distribution P (k|sk,i; ΘD, ΘS) as:

where W is a learnable parameter and b is a bias; ΘD are the parameters of discriminator; ΘS indicate the parameters of shared spaces.

Based on such adversarial structure, besides the evaluation loss Jeval, we additionally introduce an adversarial loss, so that the discriminator could help to prevent shared spaces blending with task-specific features. The adversarial loss contains two parts: one is to train the discriminator to make an accurate prediction, and the other one aims to prevent the discriminator from predicting the class of monolingual evaluation tasks.

The task discriminator learns to determine which task the feature belongs to. Thus the training objective of it is to maximize the cross entropy of predicted task distribution. The loss function is formulated as follows:

where K denotes the evaluation tasks. It updates the parameters of discriminator ΘD to minimize the loss function.

The other part of adversarial loss aims to prevent the discriminator from predicting the class of tasks. Therefore the training objective is:

which is minimized by updating the parameters of shared layers ΘS. p(k|sk,i;ΘD,ΘS) is referred as p(k|sk,i) for short.

Combining the task evaluation loss and the adversarial loss, the final loss function of our adversarial multi-task neural metric for multi-lingual dialogue evaluation is defined as:

where Jeval is computed in Eq (5).

4 Experiments

In this section, we evaluate the correlation between our proposed metrics and manual annotation, which is the ultimate goal of automatic metrics. Our model is trained with Chinese and English datasets under the adversarial multi-task neural network framework. The overall performance is investigated on Chinese and English corpus respectively.

4.1 Datasets

Chinese Corpus

We build a Chinese corpus using data crawled from an online Chinese forum Douban 2 . The training set contains 1,568,241 samples, each of which consists of a query-reply pair (in text). Standard Chinese word segmentation is applied to get Chinese terms as primitive tokens. We maintain a vocabulary of 129,506 phrases ranking by the term frequency, we empirically cut the phrases that frequency is under 3.

2 http://www.douban.com/

English Corpora

We use the Twitter Corpus 3 that contains a large number of conversations between users on the micro-blogging platform Twitter as English Corpora. The training set contains 2,537,449 query-reply pairs. Like in Chinese corpus, we maintain a vocabulary of 125,291 words, of which the frequency is higher than 5.

4.2 Implementation Details

Hyperparameters

For sentence encoder, we set the word embedding size de for both tasks to 128, and they are initialized randomly and learned during training. The Bi-LSTM hidden states dimension dh is set to 256 empirically. The learning rate α of Bi-LSTM units is initialized to 0.001. We use a two layers multi-layer perceptron to measure the relatedness of a given query and a reply. The dimension of the first layer d1m is set to 8 ∗ dh + 1 and dimension d2m of the second layer is 50. The optimizer used in both Bi-LSTM and MLP are Adam [Kingma and Ba, 2014], and the gradient is computed by standard back-propagation. We set batch size (mini-batch) of both tasks to 128, and evaluate model on dev set after every 200 steps.

Performance Evaluation

We evaluate metrics on a generative model based on sequence-to-sequence (seq2seq) neural network [Bahdanau et al., 2014]. This generative model encodes a query into a vector representation through a recurrent neural network (RNN), and decodes this vector into a reply with another RNN. To improve the performance of seq2seq model, attention mechanism is applied.

3 http://www.twitter.com/

The English and Chinese test set include 300 queries and generated replies, respectively. We had 9 volunteers to express their human satisfaction of a generated reply to a query by rating an integer score among 0, 1 and 2. Score 2 means a "good" reply, 1 borderline, and 0 bad reply.

4.3 Results and Analysis

Table 1 shows the Pearson and Spearman correlation between some metrics and human scores. The evaluated metrics are as follows.

Referenced metrics predict the score(r, ˆ r) between the ground-truth reply r and generated reply ˆ r, including BLEU, ROUGE, and Greedy Matching (GM).

Unreferenced metrics include our Single Task, Nonadversarial Multi-task (Non-ADVMT) and Adversarial Multi-task (ADVMT) neural metrics. These metrics are unreferenced, because they predict the score(q, ˆ r) between the query and its generated reply ˆ r, without referring to a groundtruth reply r.

RUBER [Tao et al., 2018] blends the referenced and unreferenced metrics by heuristics. For referenced score(r, ˆ r) and unreferenced score(q, ˆ r), it chooses the larger value (denoted as max), smaller value (min), and averaging (either geometric or arithmetic mean).

ADVMT+GM combines our Adversarial Multi-task neural metric and the Greedy Matching metric. The hybrid approach is the same as RUBER.

Overall Performance

The first observation in table 1 is that our unreferenced metrics are more correlated than those referenced metrics with human judgment in both English and Chinese evaluation tasks. This is because the referenced metrics mainly capture the similarity, but the rich semantic relationship between queries and replies necessitates more complicated mechanisms like neural networks. Besides, the referenced metrics have to rely on the information of both reference reply and generated reply, while neural network-based metrics use no reference but query and model reply. This observation shows that the query alone is also informative and that negative sampling could help to train the evaluation metrics, although it does not require human annotation as labels.

In unreferenced metrics, compared to single-task trained metric and non-adversarial multi-task trained metric, the ADVMT metric trained under the framework of adversarial multi-task achieves a best result in the correlation with human judgment. Although the corpora used in multi-task training are in different languages, the shared-private architecture shows the ability to extract useful language-invariant features. Thus when evaluating on a single language dialogue system, with the help of that shared information across languages, the performance is boosted. In addition, this result shows that incorporating the adversarial strategy could strengthen the ability to extract language-invariant features in shared spaces and help to prevent the shared spaces of features from being interfered by private spaces.

We combine the referenced metric GM and unreferenced metric ADVMT, and the hybrid approach is the same as RUBER. Experiments show that ADVMT+GM metric achieves the best result than RUBER peak performance, when choosing the geometric mean blended strategy. What's more, in both ADVMT+GM and RUBER metrics, choosing the larger value (max) is too lenient, and is slightly worse than other strategies. More importantly, our ADVMT metric is trained in multi-lingual dataset, which could be applied in multi-lingual dialogue evaluation simultaneously, while the RUBER metric should be trained in each monolingual dataset respectively and ignores massive information across languages.

Case Study

Table 2 illustrates some examples of Single Task, NonADVMT, ADVMT neural metric. As for the non-universal reply, we find that our ADVMT metric tends to give a closer score with the human score than the Single Task metric and Non-ADVMT metric. We further observe that a common problem of generative model is that it tends to generate a universal reply, such as "Ha-ha." and "Ok!" We observe that our ADVMT metric tends to give a lower score when it comes to such universal replies, while the Single Task metric and Non-ADVMT metric gives a relatively high score. Improving the diversity in generative model remains a challenging problem. But it may be furthered if the evaluation metrics used in training process encourage replies that of high diversities, and discourage those universal replies.

5 Conclusion and FutureWork

In this paper, we propose an adversarial multi-task neural metric for multi-lingual dialogue evaluation, using shared feature extraction across languages. In addition, we incorporate adversarial strategy to shared spaces, which aims to guarantee the purity of shared feature spaces. Our proposed model regards models that trained in different language corpora as a single task and integrates each single task under the framework of adversarial multi-task learning. Experiments show that the proposed model outperforms the monolingual ones and various existing metrics.

An important direction of future research is evaluating the ability of the proposed metric to transfer knowledge from one language to another. There could be a problem of lacking training corpus when it comes to the dialogue system on minority languages. As the proposed metric could extract information across languages, the performance of multilingual evaluation metrics, which are trained on some majority languages with a massive corpus, of transferring the shared knowledge to the minority languages is worth exploring.

Acknowledgments

We appreciate the contribution from Chongyang Tao. Besides, we would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2017YFC0804001), the National Science Foundation of China (No. 61672058). Rui Yan was sponsored by CCFTencent Open Research Fund and MSRA Collaborative Research Program.