Multiway Attention Networks for Modeling Sentence Pairs

1 Introduction

In this paper, we study the task of modeling a pair of sentences, which aims to compare two sentences and identify the relationship between them. It is a fundamental technology for a variety of tasks. For example, in the paraphrase identification task, it is used to determine whether two sentences are paraphrase or not [Madnani et al., 2012]. In the natural language inference task, it is utilized to judge whether a hypothesis sentence can be inferred from a premise sentence [Bowman et al., 2015]. In the answer sentence selection task, it is employed to assess the relevance between questionanswer pairs and rank all candidate answer sentences [Yang et al., 2015]. Table 1 shows corresponding examples of abovementioned three tasks.

Previous works using neural networks with attention mechanism show effectiveness on this task. These methods can be organized into two frameworks. The first framework is to model sentence pairs by encoding each sentence separately and then making the decision based on the two representations [Yu et al., 2014; Bowman et al., 2016]. The limitation ence, and the bilinear attention function which is utilized in Chen et al. [2016] to match the question and passage in the reading comprehension. In addition to these two widely-used attention functions, we use two other attention functions that calculate the word relation by the element-wise dot product and difference of two vectors. Such element-wise comparisons are conducted on distributed representations of different text granularities in previous works [Wang and Jiang, 2016a; Chen et al., 2017] to model the interactions between words and sentences. We directly apply these two functions between words, which enhances the word level relation modeling.

∗ Contribution during internship at Microsoft Research Asia.

Our model is built on the matching-aggregation framework. Given two sentences, we encode them with a bidirectional RNN to obtain contextual word representation of words in two sentences based on the word embeddings. Then we apply four above-mentioned attention functions to match two sentences at the word level. Next, we aggregate the matching information from multiway attention functions with two steps. The matching information of each function is first aggregated by a bi-directional RNN along with all words. Then the outputs of all attention functions are adaptively combined by applying the attention mechanism. We apply another bi-directional RNN to pass through the combined representation to aggregate the mixture matching information. Finally, we apply the attention-pooling to the matching information for a fix-length vector and feed it into a multilayer perceptron for the final decision.

We conduct experiments on three tasks with four standard benchmark datasets. Experimental results show that our multiway attention network achieves state-of-the-art results on the Quora Question Pairs dataset for paraphrase identification, the SNLI and MultiNLI datasets for natural language inference, and the SQuAD dataset for answer sentence selection. We also conduct ablation tests by using single attention function and removing any of functions. Results justify that all of four attention functions help the model performance. Moreover, we observe that different attention functions are good at samples of different categories, which explains why our framework improves the result since it combines the strengths frommultiple attention functions.

2 RelatedWork

Recently, deep learning and neural network models achieve promising results in modeling sentence pairs. Basically, two sentences are encoded into sentence vectors with a neural network encoder, and then the relationship between two sentences was decided solely based on the two sentence vectors [Bowman et al., 2015; Yang et al., 2015; Tan et al., 2016]. However, this kind of framework ignores the lower level interaction between two sentences.

The matching-aggregation framework is therefore proposed matching two sentences at the word level and then aggregating the matching information based on the attention mechanism for the final decision. Rockt¨aschel et al. [2015] use the attention-based technique to improve the performance of LSTM-based recurrent neural network. They employ the word-by-word attention to obtain a sentence-pair encoding from fine-grained reasoning via soft-alignment of words and phrases in the premise and hypothesis, which achieves very promising result on the SNLI dataset. Wang and Jiang [2016b] propose match-LSTM for the natural language inference that tries to match the current word in the hypothesis with an attention-weighted representation of the premise calculated by the word-by-word attention. Wang et al. [2017b] propose BiMPM that matches two sentences by a bilateral matching with attention mechanism in multiple perspectives, which achieves the state-of-the-art results on the paraphrase identification, natural language inference, and answer sentence selection. They propose four types of representation instead of the attention-weighted representation to improve the result. Moreover, Cheng et al. [2016] propose LSTMN, which improves the attention mechanism by a memory network for the natural language inference. Parikh et al. [2016] match two sentences by attending, comparing, and aggregating for the SNLI dataset. Sha et al. [2016] propose re-reading two sentences with the attention mechanism to improve the memory of the other sentence for a better understanding. Munkhdalai and Yu [2016] propose using the tree structure to improve the recurrent or recursive architecture for the natural language inference and answer sentence selection. Chen et al. [2017] incorporate syntactic parsing information to basic attention-based neural networks, which achieves the state-of-the-art result on the SNLI dataset.

Furthermore, CNN is also applied on the sentence pairs modeling and achieves very promising results [Gong et al., 2017]. Yin et al. [2016] propose the attention-based CNN for the paraphrase identification, natural language inference, and answer sentence selection. Wang and Jiang [2016a] use the CNN to aggregate the matching information for the answer sentence selection.

Our model belongs to the matching-aggregation framework. Based on the attention mechanism, we propose using multiple functions to match words in two sentences, and make the decision by aggregating the matching information from multiple attention functions.

3 Task Definition

Formally, we can represent each example of sentence pairs as a triple (Q, P, y), where Q = (q1, ..., qi, ..., qN) is a sentence with a length N, P = (p1, ..., pj , ..., pM) is another sentence with a length M, and y ∈ Y is the label representing the relationship between Q and P .

Specifically, for a paraphrase identification task, Q and P are two sentences, Y = {0, 1}, where y = 1 means that Q and P are paraphrase of each other, and y = 0 otherwise.

For a natural language inference task, Q is a premise sentence, P is a hypothesis sentence, and Y ={entailment, contradiction, neutral} where entailment indicates P can be inferred fromQ, contradiction indicates P cannot be true condition on Q, and neutral means P and Q are irrelevant to each other.

For an answer sentence selection task, Q is a question, P is a candidate answer sentence, and Y = {0, 1} where y = 1 means P is a correct answer sentence for Q, and y = 0 otherwise.

dings and contextual embeddings. The contextual embeddings are generated by taking the output of a pre-trained language model [Peters et al., 2018], which has shown the effectiveness on many NLP tasks. We then use a bidirectional GRU to produce new representation hq1 , . . . , hqN and np1 , . . . , hPM of all words in two sentences respectively:

and hqt = [-→h qt , ←- h qt ]. Meanwhile, we use another bidirectional GRU to encode each wp t to the representation hpt .

4.3 MultiwayMatching

Previous works show the effectiveness of the word-level attention in modeling sentence pairs [Rockt¨aschel et al., 2015; Wang and Jiang, 2016b]. In our model, we design multiple attention functions to compare two vectors of the words in two sentences, namely concat attention, bilinear attention, dot attention, and minus attention. Therefore, at each position t of P, the word can match Q using four attention functions to obtain corresponding weighted-sum representations of Q.

where hpt is the representation of P at the position t, hq is the representation of all positions in Q, and qk t is the corresponding representation of Q using the attention function fk based on the parameter Wk, where k = (c, b, d,m), which represents the concat attention, bilinear attention, dot attention, and minus attention, respectively.

4 Our Approach

We show the multiway attention networks in Figure 1. It consists of five parts under the matching-aggregation framework. Specifically, we obtain contextual word representation for two sentences using a bi-directional RNN. Then we match words between two sentences in multiple ways. For every word pair from Q and P , we can obtain four matching scores using multiple attention functions. Next, we aggregate the matching information along with words in P in two steps. We first match two sentences inside each attention function, and then combine the matching information from all functions. The bi-directional RNN is applied to aggregate the matching information both in the inside aggregation and mixed aggregation. Finally, we use the attention pooling to aggregate the matching information in P for a fix-length vector and apply a multilayer perceptron (MLP) classifier for the final decision.

4.1 Gated Recurrent Unit

We use Gated Recurrent Unit (GRU) [Cho et al., 2014] instead of basic RNN. Equation 1 describes the mathematical model of the GRU. rt and zt are the gates and ht is the hidden state.

4.2 Encoding Layer

Consider two sentences Q = {wq t }Nt=1 and P = {wp t }Mt=1 , we first convert the words to their respective word-level embed-

Concat Attention :

Bilinear Attention :

Dot Attention :

Minus Attention :

stj = vmT tanh(Wm(hqj - hpt ))

(7a)

4.4 Aggregation

We aggregate the matching information from multiway attention functions in two steps. The inside aggregation is to aggregate the matching information along with words in the sentence P inside each attention function. For each position t, we concatenate the word representation hpt in P with its corresponding representation qk t of Q [Wang and Jiang, 2016b] and add a gate to determine the importance of the concatenated representation [Wang et al., 2017a]. Then we use the bi-directional GRU to pass through each position in P. We take the concat attention as an example,

and hct = [-→h ct , ←- h ct ]. For the bilinear, dot, and minus attention, we obtain the hbt , hdt , and hmt , respectively.

The mixed aggregation is to combine the matching information from all attention functions. We apply the attention mechanism to adaptively combine four representations with the parameter va as the input.

Then, we feed xt into a bi-directional GRU to aggregate the matching information from multiple attention functions. We obtain hot = [-→h ot , ←- h ot ] for all positions in P.

4.5 Prediction Layer

After aggregating the information from multiway matching, we convert the resulting representations of all positions in P to a fixed-length vector with pooling and feed it into a classifier to determine the relation between two sentences. We first apply an attention pooling with the parameter vq to select the important information in Q.

Then we use this representation rq to select the information in the matching vectors.

Finally, we feed rp into a multilayer perceptron (MLP) classifier for the probability pi of each label in the corresponding task.

For all tasks, the objective function is to minimize the following cross entropy:

where yi denotes a label, in paraphrase detection it is (0, 1) to represent whether two sentences are paraphrase, in natural language inference it is the relation of two sentences of entailment, contradiction, and neutral, in answer sentence selection it is (0, 1) to represent whether P can answer the question Q.

4.6 Implementation Details

We use 300-dimensional uncased pre-trained GloVe embeddings [Pennington et al., 2014] without update during training. We use zero vectors to represent all out-of-vocabulary words. Hidden vector length is set to 150 for all layers. We apply dropout [Srivastava et al., 2014] between layers, with dropout rate 0.2. The model is optimized using AdaDelta [Zeiler, 2012] with initial learning rate of 1.0.

5 Experiment

We conduct experiments on three tasks with four datasets. Experimental results show that our model outperforms our baseline and other competing approaches. We also conduct ablation tests to analyze the contribution of each attention function.

5.1 Dataset and EvaluationMetrics

Quora Question Pairs This dataset consists of over 400,000 question pairs, and each question pair is annotated with a binary value indicating whether the two questions are paraphrase of each other. We select 5,000 paraphrases and 5,000 non-paraphrases as the development set, and use another 5,000 paraphrases and 5,000 non-paraphrases as the test set. We keep the remaining instances as the training set.

SNLI It is a natural language inference dataset [Bowman et al., 2015]. The original data set contains 570,152 sentence pairs, each labeled with one of the following relationships: entailment, contradiction, neutral and -, where - indicates a lack of human annotation and is usually discarded [Wang and Jiang, 2016b]. In the end, we have 549,367 pairs for training, 9,842 pairs for development and 9,824 pairs for test.

MultiNLI It is a natural language inference dataset [Williams et al., 2017]. Models can be evaluated on both the matched test examples, which are derived from the same sources as those in the training set, and the mismatched examples, which do not closely resemble any seen at training time. This dataset contains 392,702 pairs for training, 9,815 matched pairs and 9,832 mismatched pairs for development, 9,796 matched pairs and 9,847 mismatched pairs for test.

SQuAD It is a reading comprehension dataset, where the answer to each question is a span of text from the corresponding passage [Rajpurkar et al., 2016]. To evaluate our answer selection task, we split the sentences from the passage using the Stanford CoreNLP Toolkit [Manning et al., 2014] and treat the sentence which contains the correct span as the answer sentence. As the author of SQuAD only publishes the training set and the development set, we split the 10,570 instances in the development set to 5,000 for development and 5,570 for test. We use mean reciprocal rank (MRR) for evaluation.

5.2 Result on Paraphrase Identification

We compare our model with several baselines shown in Table 2. Siamese-CNN and Siamese-LSTM [Wang et al., 2016a] are based on the sentence encoding framework. Multi-Perspective-CNN and Multi-Perspective-LSTM improve them by multiple perspective cosine matching functions [Wang et al., 2017b]. We also compare our MwAN with models under the matching-aggregation framework, such as LDC [Wang et al., 2016b] and BiMPM [Wang et al., 2017b] under RNN and DIIN [Gong et al., 2017] under CNN. Our MwAN outperforms all baselines with 89.60% accuracy on the development set and 89.12% accuracy on the test set.

5.3 Result on Natural Language Inference

Table 3 shows results on the SNLI. Our MwAN achieves the state-of-the-art result with 88.3% test accuracy. Moreover, we also report the ensemble result. Following the ensemble strategy in Wang et al. [2017b], we train the model 4 times with the same setting, and sum the probability of each single model to decide the result. The test accuracy is 89.4%, which outperforms all baselines and achieves the best test score on the SNLI benchmark to date.

Table 4 shows results on the MultiNLI. Results of baselines are taken from the official report [Williams et al., 2017]. Our MwAN achieves the state-of-the-art results of 78.5% and 77.7% on the Matched and Mismatched pairs, respectively. An ensemble model achieves the state-of-the-art result with 79.8% accuracy on the Matched pairs, and outperforms all baselines with 79.4% accuracy on the Mismatched pairs.

5.4 Result on Answer Sentence Selection

For the answer sentence selection task, we do not use the WikiQA dataset because it is too small to train a complex deep learning model. We implement two basic deep learning baselines Bigram-CNN [Yu et al., 2014] and Attentive LSTM [Tan et al., 2016], and we also report results of two state-of-the-art methods BiMPM [Wang et al., 2017b] and CNN-MULT [Wang and Jiang, 2016a]. All these baselines achieve very promising results on the WikiQA dataset for this task. As shown in Table 6, our MwAN achieves 91.35 in terms of MRR, which outperforms all baseline methods.

5.5 Discussion

To analyze the effectiveness of each attention function, we conduct ablation tests on four datasets by using single attention function and removing any of attention functions. Since the label of test data in MultiNLI is not published, we analyze it using the Matched and Mismatched data in the development set. As illustrated in Table 5, our MwAN is obviously better than using single attention function. In addition, removing any of attention functions leads to the worse score, which shows the effectiveness of our multiway attention network.

Next, we analyze the effect of four attention functions in different categories of samples. In our adaptive combinations, we can obtain the weight of each attention function at every position in P (Equation 9b). We average all weights of each attention function. Table 7 shows the result for four attention functions. We observe that each attention function has different effect on samples in different categories. The weight of the dot attention increases in positive samples (paraphrase in Quora, entailment in SNLI, and answer sentence in SQuAD) and the other functions increase in negative samples (nonparaphrase in Quora, contradiction in SNLI, and non-answer sentence in SQuAD), which indicates the dot attention is dominant in modeling similarity and other functions are good at modeling difference. Our multiway attention network combines their strengths, therefore achieves better results.

6 Conclusion

In this paper, we propose a multiway attention network that applies multiple attention functions to model the matching between a pair of sentences. We mainly focus on three tasks, namely paraphrase identification, natural language inference, and answer sentence selection. After obtaining contextual word representation based on the word embeddings. We propose using four different attention functions to match words in two sentences. Next, we aggregate the matching information from multiway attention functions. Finally, we use the attention pooling to aggregate the matching information into a fix-length vector and feed it into a classifier for the final decision. Experimental results show that our proposed multiway attention network outperforms baseline neural network models and yields state-of-the-art results on the Quora Question Pairs for the paraphrase identification, SNLI and MultiNLI for the natural language inference, and the answer sentence selection task on the SQuAD dataset.

Acknowledgments

We greatly thank Nan Yang for helpful discussions. We also thank all the anonymous reviewers for their helpful comments. Chuanqi Tan and Weifeng Lv are supported by the National Key R&D Program of China (No. 2017YFB1400200) and National Natural Science Foundation of China (No. 61421003 and 71501003).