Attention-Fused Deep Matching Network for Natural Language Inference

1 Introduction

Natural language inference (NLI) is a core challenge in the natural language processing community, which aims to predict whether a premise sentence can infer another hypothesis sentence [MacCartney, 2009]. This problem is usually viewed as a classification problem. Existing approaches for NLI are categorized into two types: conventional discrete feature-based approaches [Bowman et al., 2015] and neural network-based models [Chen et al., 2017a].

Neural network-based models have attracted more attention for their ability in assisting efforts in feature engineering [Wang and Jiang, 2015; Chen et al., 2017a; Wang et al., 2017b]. Despite their success, there are still two problems.

The first problem is long-term context dependency. Context dependency is essential for NLI, but is difficult to model, especially long-term context dependency. Most previous models [Bowman et al., 2015; Liu et al., 2016b] who focus on modeling the contextual information adopt a long shortterm memory network (LSTM) [Hochreiter and Schmidhuber, 1997] or a gated recurrent units network (GRU) [Chung et al., 2014]. However, there exist in NLI some sentences (L ≥ 17) whose long-term dependency cannot be effectively modeled by LSTM and GRU because of their length.

∗Contribution during internship at Microsoft Research Asia.

The second problem is insufficient model complexity. Recent evidence [Simonyan and Zisserman, 2014; Szegedy et al., 2015] reveals that network depth is of crucial importance. Previous neural network-based models can be categorized into two classes: sentence encoding based models and attention-based models. For the first class, Bowman et al. [2015], Tan et al. [2015] propose matching models that use the sentence vectors. This method is simple and effective, but neglects the interaction between two sentences. For the second class, Wang et al. [2017b]; Chen et al. [2017a] have designed new matching models that leverage attentions from two directions, where bidirectional information is added to achieve higher accuracy. However, the network structures of previous models are still so shallow that the model capability is not sufficient for modeling complex relations.

In order to address these two problems, we propose an attention-fused deep matching network (AF-DMN) for the NLI task. AF-DMN is a neural network structure stacked with multiple computational blocks in its matching layer. Each computational block consists of four sub-layers: (1) a cross attention layer; (2) a fusion layer for cross attention; (3) a selfattention layer; and (4) another fusion layer for self-attention. In this model, we utilize two heterogeneous attention mechanisms in each computational block: cross attention aims to make information interaction between two sentences, while self-attention aims to exploit long-term context dependency. In addition, we employ fusion layers to refine the representation following the two attention layers respectively. Experiments on the SNLI, the MultiNLI and the Quora duplicate questions datasets demonstrate that AF-DMN significantly improves accuracy and achieves state-of-the-art performance. Our contributions are summarized as follows:

• The AF-DMN model incorporates two attention mechanisms jointly: cross attention aims to make information interaction between two sentences; while self-attention aims to exploit long-term context dependency.
• We first conduct multiple stacked computational blocks in the matching layer for NLI, which allows the model

to learn the interaction of the sentence pair better.

• We evaluate our model on three challenging datasets and the results show that our model outperforms state-of-theart baselines.

2 General Neural Model for Natural Language Inference

NLI is usually regarded as a classification task that predicts the relation y ∈ Y for a given pair of sentences, where Y = {entailment, contradiction, neutral}. Recently, neural networks have been widely applied to this task because of their ability to assist with feature engineering [Bowman et al., 2015; Wang and Jiang, 2015; Wang et al., 2017b].

Formally, given two sentences p = (p1, ..., pi, ..., pm) and q = (q1, ..., qj, ..., qn), the aim is to predict the ground truth of relation y∗ :

The Pr(·) is a neural network here. The general architecture of neural networks for NLI consists of three components: (1) an encoder layer converts the two sentences into semantic representations; (2) a matching layer aligns the information between the two sentences at the word level and produces new representations for the two sentences; and (3) a prediction layer predicts the relation of the given pair of sentences.

2.1 Encoder Layer

Given two sentences p = (p1, ..., pi, ..., pm) and q = (q1, ..., qj, ..., qn), the encoder layer first converts them into vectors (ep1 , ..., epi , ..., epm ) and (eq1 , ..., eqi , ..., eqn ) by looking up M respectively, where M ∈ Rd×|V | is the embedding table. d is the dimension of embeddings and |V | is the size of the vocabulary. The encoder layer then produces the semantic representation for each word in p and q using the Bi-directional Long Short-Term Memory (BiLSTM) neural network [Hochreiter and Schmidhuber, 1997]. Thus, each word in two sentences can be expressed as:

where, hp0 and hq0 are initialized as 0. Thus, the two sentences are converted to Hp = (hp1 , ..., hpi , ..., hpm ) and Hq = (hq1 , ..., hqj , ..., hqn ).

2.2 Matching Layer

Generally, the matching layer interacts with the information between two sentences for alignment. It can be formulated as:

where g(·) is the matching function. Thus, Vp = (vp1 , ..., vpi , ..., vpm ) and Vq = (vq1 , ..., vqj , ..., vqn ) are the new representations for p and q respectively.

2.3 Prediction Layer

In this layer, a pooling layer is used to convert the vectors into a fixed-length vector and then feed it into a 2-layer multi-layer perception (MLP) classifier.

In order to capture all of the information and highlight the significant properties of the two sentences, we perform a mean pooling and a max pooling on each of them and then concatenate them together:

After obtaining the representation V of the two sentences, the distribution Pr(·) can be formalized as:

where W1, W2, b1, and b2 are trainable parameters.

3 Attention-Fused DeepMatching Network for Natural Language Inference

Despite the success of conventional NLI models [Wang and Jiang, 2015; Chen et al., 2017a; Wang et al., 2017b], they only generate a shallow interaction between two sentences, which cannot sufficiently model the complex semantic interactions in the inference problem.

Inspired by the recent successful deep neural frameworks [He et al., 2016; Wu et al., 2016], we propose AF-DMN for natural language inference. AF-DMN exploits a more sophisticated matching layer, which consists of T computational blocks and each block has four sub-layers: (1) a cross attention layer; (2) a fusion layer for cross attention; (3) a selfattention layer; and (4) another fusion layer for self-attention. The AF-DMN will repeat the interaction process via stacked computational blocks T times.

3.1 Cross Attention

Cross attention captures the relevance between two sentences p and q. Concretely, in the t-th computational block, given the representations of two sentences computed in the previous block: Ht-1 p = (ht-1 p1 , ..., ht-1 pi , ..., ht-1 pm ) and Ht-1 q = (ht-1 q1 , ..., ht-1 qj , ..., ht-1 qn ), we first compute a co-attention matrix At ∈ Rm×n . Each element Ati,j ∈ R indicates the relevance between the i-th word of sentence p and the j-th word of sentence q. Formally, the co-attention matrix could be computed as:

where Wt ∈ R2h×2h , Utl ,Utr ∈ R2h are for the t-th computational block and 〈·, ·〉 denotes the inter production operation. Then the attentive representation for each word pi and qj could be formalized as ˜htpi ∈ R2h and ˜htqj ∈ R2h :

3.2 Fusion for Cross Attention

In order to enhance the interaction further, we perform a fusion layer after the cross attention layer:

where [·; ·; ·; ·] refers to the concatenation operation. Similarly, we derive the fusion result for sentence q as f t qj .

3.3 Self-Attention

In order to tackle the long-term dependency in a long sentence, we additionally introduce the self-attention mechanism after the cross attention layer.

Formally, for sentence p, we first compute a self-attention matrix St ∈ Rm×m:

where Sti,j indicates the relevance between the i-th word and j-th word in sentence p.

Then the self attentive vector for each word can be computed as follow:

where Ftp = (f tp1 , ..., f tpi , ..., f tpm ), and f tpi is computed as in Eq. (14).

We can similarly derive the self attentive vector for sentence q as ¯htqj .

3.4 Fusion for Self-Attention

A fusion layer is introduced after the self-attention to enhance interaction. The fusion representation htpi of sentence p will be sent to the next block of the matching layer. Formally, htpi is computed as:

Similarly, we obtain the representation htqj for sentence q. Thus, in the t-th computational block, two sentences are converted to Htp = (htp1 , ..., htpi , ..., htpm ) and Htq = (htq1 , ..., htqj , ..., htqn ). Finally, Htp and Htq are sent to the prediction layer as input Vp and Vq after conducting the matching process T times.

4 Training

The object is to minimize the objective function J(Θ), which can be formulated as:

where N is the number of instances in the training set and (p(i) , q(i) ) and y(i) are the sentence pair and the corresponding annotated label for the i-th instance respectively. Θ denotes all the trainable parameters of our model. We employ Adam [Kingma and Ba, 2014] as the optimizer.

5 Experiments

5.1 Dataset

We evaluate our model on three datasets: the Stanford Natural Language Inference (SNLI), the MultiGenre NLI Corpus (MultiNLI) and Quora duplicate questions 1 (Quora). The detailed statistical information of datasets is shown in Table 1.

SNLI The SNLI corpus [Bowman et al., 2015] contains 570,152 sentence pairs. Each pair is labeled with one of the following relationships: entailment, contradiction, or neutral. The data partition follows that of [Bowman et al., 2015].

MultiNLI The MultiNLI corpus [Williams et al., 2017] is a new dataset for NLI, which contains 433k sentences pairs. Similar to SNLI, each pair is labeled with one of the following relationships: entailment, contradiction, or neutral. Since the MultiNLI corpus is collected from multiple domains, there are in-domain and cross-domain development/test sets.

Quora The Quora corpus contains over 400,000 question pairs. Each question pair is labeled with a binary value indicating whether the two questions are paraphrases of each other. In our experiment, we have the same partition as in [Wang et al., 2017b].

5.2 Experiment Configuration

In our model, word embeddings and all hidden states of LSTMs and MLPs are 300 dimensions. For the SNLI dataset, there are 3 computational blocks in the deep matching layer, while there are 2 for MultiNLI and Quora datasets. We employ the Adam [Kingma and Ba, 2014] for training, whose default hyper-parameters β1 and β2 are set to 0.9 and 0.999 for optimization respectively. The initial learning rate of Adam is set to 0.0002. The learning rate is halved when the accuracy on the development set drops. We also employ a dropout strategy [Srivastava et al., 2014] on word embeddings and all MLPs to avoid over-fitting. The dropout rate is set to 0.2. The batch size is set to 64. We set the maximum length of sentences to 200. For preprocessing, we just tokenize the sentences and lowercase the tokens.

For initialization, word embeddings are initialized with 300-dimensional GloVe vectors and updated during training. Other parameters include neural network parameters and Out Of Vocabulary (OOV) word embeddings are initialized randomly within [-0.01,0.01].

1 https://data.quora.com/First-Quora-Dataset-Release-QuestionPairs

5.3 Ensemble

The ensemble strategy is an effective method to improve model accuracy. Following [Wang et al., 2017b], our ensemble model averages the probability distributions from three individual single AF-DMNs, who have exactly identical architectures but distinguished initializations on parameters.

5.4 Overall Results

We use the accuracy to evaluate the performance of AF-DMN and other models on datasets SNLI, MultiNLI, and Quora.

SNLI Table 2 shows the results of different models on the training and test sets of SNLI. The baseline models in Table 2 can be categorized into two groups:

(1) The first group of models are based on sentence encoding. The TBCNN-pair neural model [Mou et al., 2015] incorporates structural information into sentence representation. SPINN-PI [Bowman et al., 2016] integrates tree-structured sentence interpretation into the linear sequential structure of a shift-reduce parser. Liu et al. [2016b] introduce the innerattention that using a preliminary sentence representation to attend to words within the sentence. In [Munkhdalai and Yu, 2016], a memory augmented neural network is presented.

(2) The second group are based on attention-based models. Rockt¨aschel et al. [2015] extend the general sentence encoding model with attention while Wang and Jiang [2015] exploit long short-term memory (LSTM) for NLI. Liu et al. [2016a] pay more attention to the interaction of the text pair. Parikh et al. [2016] use attention to decompose the problem into subproblems that can be solved separately. Similar to [Wang and Jiang, 2015], Sha et al. [2016] design a new LSTM unit that takes the attention vector of one sentence as an inner state while reading the other sentence. Recently, Chen et al. [2017a] incorporate the chain LSTM and tree LSTM. Wang et al. [2017b] propose a bilateral multi-perspective matching.

In Table 2, the first three blocks are single models and the last two blocks are ensemble models. The proposed models, the single AF-DMN and the ensemble AF-DMN, achieve 88.6% and 89.0% on accuracy in SNLI test set respectively. Compared to previous work, AF-DMN outperforms previous models on both single and ensemble scenarios for natural language inference.

MultiNLI Table 3 shows the performance of different models on MultiNLI. Since this dataset aims to evaluate the quality of sentences representations, typical attention and memory models are not eligible for inclusion in this competition. As a result, most existing work on MultiNLI does not use the attention mechanism except for ESIM Chen et al. [2017b]. So far, ESIM is the strongest baseline which achieves a state-ofart performance on SNLI task. As [Chen et al., 2017b] report, ESIM achieves 76.8% and 75.8% on the in-domain and crossdomain test sets of MultiNLI on accuracy respectively. The proposed model, AF-DMN, achieves 76.9% and 76.3% on the in-domain and cross-domain test sets on accuracy respectively. The results show that our model outperforms ESIM on both in-domain and cross-domain test sets.

Quora Table 4 shows the performance of different models on the Quora test set. The baselines on Table 4 are all implemented in [Wang et al., 2017b]. The Siamese-CNN model and Siamese-LSTM model encode sentences with CNN and LSTM respectively, and then predict the relationship between them based on the cosine similarity. Multi-Perspective-CNN and Multi-Perspective-LSTM are transformed from SiameseCNN and Siamese-LSTM respectively by replacing the cosine similarity calculation layer with their multi-perspective cosine matching function. The L.D.C is a general "compareaggregate" framework that performs word-level matching followed by an aggregation of convolution neural networks. As we can see, AF-DMN outperforms the baselines and achieves 88.72% in the test sets of the Quora corpus.

5.5 Effect of Components

To better understand the performance of AF-DMN, we analyze the effect of each key component of the proposed model.

As illustrated in table 5, the first row is the AF-DMN without fusion layers and self-attention mechanism (only keeping cross attention layer) that we consider as the basic model. Compared to the full AF-DMN, the accuracy drops by 3.9% on test set of the SNLI dataset. By adding the fusion layer following the cross attention layer, the performance increases to 88.2%. That is a significant improvement and it is because the information from two sentences are gathered through the fusion layer. By additionally employing the self-attention mechanism, we achieve further improvement in accuracy. Finally, applying the fusion after the self-attention layer obtains the final accuracy. According to the results, all of the components positively contribute to the final performance.

Table 6 shows the performance with a different number of blocks. As we can see, with the number of blocks increases from 1 to 3, the performance increases both on the development set and the test set. Because of computational cost, we just set the number of blocks as 3 on SNLI.

5.6 Case Study

As depicted in Figure 2, this is an instance from the test set of the SNLI dataset: {p: a person in a blue plaid shirt is writing on a chalk board. q: the person is writing on the chalk board. The label y: entailment.}. The results are produced by AF-DMN with 3 computational blocks in the deep matching layer, demonstrating the changes of cross attention from low block to high block in the matching layer. The three subfigures from left to right show the alignment results in the 1st, 2nd, 3rd computational block of the matching layer respectively. With the increment of interaction, the alignment of the sentence pair becomes more clear and accurate. The highlighted cells imply that cross attention aims to figure out the alignments between the two sentences.

Figure 3 is the visualization of the self-attention layer in the last block for the same test instance. The left sub-figure refers to sentence p and the right sub figure refers to sentence q. In the left sub figure, the phrase "in a blue plaid shirt" is highlighted, which is the inconsistent part between the two sentences and also the critical factor that determines if p can entail q. It shows that the proposed AF-DMN is capable of capturing key information between a pair of sentences for natural language inference.

5.7 Effect of Sentence Length

Since ESIM is currently the state-of-the-art model for the SNLI dataset and the code is available 2 , we would like to compare our model with ESIM on the SLNI dataset in this sub sections.

Sentence length is one of the most important factors that can affect the performance of neural models. To further analyze the performance of the proposed model, we evaluate the performance of AF-DMN and ESIM on sentence pairs of different text lengths on the SNLI test set. Specifically, we use the floor average length of a pair of sentences (p and q) as the length of the sentence pair. As illustrated in Figure 4, for sentence pairs with a length less than or equal to 8, both of the models achieve the best performance that is better than 90%. However, with the incremental increases in length of the sentence pairs, the performance of both models decreases. Throughout the results, except for sentence pairs with a length range from 9 to 10, AF-DMN is defeated with a small gap, the proposed model outperforms ESIM on rest length. When the length of sentence pair is greater than or equal to 17, our model outperforms ESIM by a large margin on this length. It indicates that the proposed model benefits from the self-attention mechanism that has an advantage in incorporating local context information within a sentence. Meanwhile, the deep matching process is able to extract key information from sentences multiple times, thereby benefiting cases on long sentences.

2 https://github.com/lukecq1231/nli

6 RelatedWorks

Natural language inference (NLI) has been widely investigated for many years. Previous work on NLI relies on handcrafted features such as n-gram overlapping, syntactic information and so on. Heilman and Smith [2010] propose an effective tree edit approach to model relations between sentence pairs. Bowman et al. [2015] adopt a lexicalized classifier which implements features (BLEU score, n-gram overlap, etc.) for SNLI. All above mentioned methods perform reasonably well for a specific task but are difficult to generalize for others.

Benefiting from the development of deep learning and the availability of large-scale annotated datasets [Bowman et al., 2015; Williams et al., 2017], data-driven models attract more attentions. Liu et al. [2016b] introduce innerattention using a preliminary sentence representation to attend words within the sentence. In [Munkhdalai and Yu, 2016], a memory augmented neural network for NLI is presented. All of these models are based on sentence encoding. However, they neglect the interaction between sentences. To address the problem, Wang and Jiang [2015] design a special LSTM called matching-LSTM which performs wordby-word matching of the hypothesis with the premise. Furthermore, Cui et al. [2016]; Wang et al. [2017b]; Chen et al. [2017a] propose a new framework to model the relationship between two sentences, which performs the matching on pairs of sentences in two directions. Besides cross attention, because of the limitations of the RNN model on the long-term dependency problem, the self-attention mechanism is proposed, which aims to align the sequence with itself and has been used in variety of tasks [Lin et al., 2017; Wang et al., 2017a]. The self-attention mechanism can capture contextual information from the whole sentence.

For the proposed AF-DMN, it performs cross attention and self-attention on a sentence pair for multiple iterations through the stacked computational blocks. Based on this mechanism, we obtain representations of two sentences with multiple levels of abstraction and achieve better performance on several challenging datasets.

7 Conclusions and FutureWork

In this paper, we propose an attention-fused deep matching network (AF-DMN) for natural language inference. It leverages cross attention and self-attention jointly. We evaluate our model on three datasets: SNLI, MultiNLI, and Quora duplicate questions. Experiment results show that AF-DMN achieves state-of-the-art performance. In the future, we will further investigate whether unlabeled data can help to learn more accurate sentence representations and relationships between inputs, to ameliorate data sparseness.

Acknowledgments

We are grateful to the anonymous reviewers. The work of this paper is funded by the (key) project of National Natural Science Foundation of China (No. 91520204, 61572154) and the project of National High Technology Research and Development Program of China (863 Program) (No. 2015AA015405). This project is also partially funded by Microsoft Research Asia.