---
title: "Bilateral Multi-Perspective Matching for Natural Language Sentences"
authors:
  - "Zhiguo Wang"
  - "Wael Hamza"
  - "Radu Florian"
doi: "10.24963/ijcai.2017/579"
url: "https://ijcai.org/proceedings/2017/579"
pdf: "https://ijcai.org/proceedings/2017/0579.pdf"
venue: "Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence"
year: 2017
pages: "4144-4150"
track: "Main track"
keywords:
  - "Natural Language Processing: Information Retrieval"
  - "Natural Language Processing: Natural Language Processing"
  - "Natural Language Processing: Natural Language Semantics"
  - "Natural Language Processing: Question Answering"
---

# Bilateral Multi-Perspective Matching for Natural Language Sentences

Zhiguo Wang, Wael Hamza, Radu Florian

*Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence*, pages 4144-4150. DOI: [10.24963/ijcai.2017/579](https://doi.org/10.24963/ijcai.2017/579)

## Abstract

Natural language sentence matching is a fundamental technology for a variety of tasks. Previous approaches either match sentences from a single direction or only apply single granular (word-by-word or sentence-by-sentence) matching. In this work, we propose a bilateral multi-perspective matching (BiMPM) model. Given two sentences P and Q, our model first encodes them with a BiLSTM encoder. Next, we match the two encoded sentences in two directions P against Q and P against Q. In each matching direction, each time step of one sentence is matched against all time-steps of the other sentence from multiple perspectives. Then, another BiLSTM layer is utilized to aggregate the matching results into a fix-length matching vector. Finally, based on the matching vector, a decision is made through a fully connected layer. We evaluate our model on three tasks: paraphrase identification, natural language inference and answer sentence selection. Experimental results on standard benchmark datasets show that our model achieves the state-of-the-art performance on all tasks.

## Full text

## 1 Introduction

Natural language sentence matching (NLSM) is the task of comparing two sentences and identifying the relationship between them. It is a fundamental technology for a variety of tasks. For example, in a paraphrase identification task, NLSM is used to determine whether two sentences are paraphrase or not [Yin et al., 2015]. For a natural language inference task, NLSM is utilized to judge whether a hypothesis sentence can be inferred from a premise sentence [Bowman et al., 2015]. For question answering and information retrieval tasks, NLSM is employed to assess the relevance between query-answer pairs and rank all the candidate answers [Wang et al., 2016d]. For machine comprehension tasks, NLSM is used for matching a passage with a question and pointing out the correct answer span [Wang et al., 2016b].

With the renaissance of neural network models [LeCun et al., 2015; Peng et al., 2015a; 2016], two types of deep learning frameworks were proposed for NLSM. The first framework is based on the "Siamese" architecture [Bromley et al., 1993]. In this framework, the same neural network encoder (e.g., a CNN or a RNN) is applied to two input sentences individually, so that both of the two sentences are encoded into sentence vectors in the same embedding space. Then, a matching decision is made solely based on the two sentence vectors [Bowman et al., 2015; Tan et al., 2015]. The advantage of this framework is that sharing parameters makes the model smaller and easier to train, and the sentence vectors can be used for visualization, sentence clustering and many other purposes [Wang et al., 2016c]. However, a disadvantage is that there is no explicit interaction between the two sentences during the encoding procedure, which may lose some important information. To deal with this problem, a second framework "matchingaggregation" has been proposed [Wang and Jiang, 2016; Wang et al., 2016d]. Under this framework, smaller units (such as words or contextual vectors) of the two sentences are firstly matched, and then the matching results are aggregated (by a CNN or a LSTM) into a vector to make the final decision. The new framework captures more interactive features between the two sentences, therefore it acquires significant improvements. However, the previous "matchingaggregation" approaches still have some limitations. First, some of the approaches only explored the word-by-word matching [Rockt¨aschel et al., 2015], but ignored other granular matchings (e.g., phrase-by-sentence); Second, the matching is only performed in a single direction (e.g., matching P against Q) [Wang and Jiang, 2015], but neglected the reverse direction (e.g., matching Q against P ).

In this paper, to tackle these limitations, we propose a bilateral multi-perspective matching (BiMPM) model for NLSM tasks. Our model essentially belongs to the "matchingaggregation" framework. Given two sentences P and Q, our model first encodes them with a bidirectional Long ShortTerm Memory Network (BiLSTM). Next, we match the two encoded sentences in two directions P → Q and P ← Q. In each matching direction, let's say P → Q, each time step of Q is matched against all time-steps of P from multiple perspectives. Then, another BiLSTM layer is utilized to aggregate the matching results into a fixed-length matching vector. Finally, based on the matching vector, a decision is made through a fully connected layer. We evaluate our model on three NLSM tasks: paraphrase identification, natural language inference and answer sentence selection. Experimental results on standard benchmark datasets show that our model achieves the state-of-the-art performance on all tasks.

In following parts, we start with a brief definition of the NLSM task (Section 2), followed by the details of our model (Section 3). Then we evaluate our model on standard benchmark datasets (Section 4). We talk about related work in Section 5, and conclude this work in Section 6.

## 2 Task Definition

Formally, we can represent each example of the NLSM task as a triple (P,Q, y), where P = (p1, ..., pj, ..., pM) is a sentence with a length M, Q = (q1, ..., qi, ..., qN) is the second sentence with a length N, y ∈ Y is the label representing the relationship between P and Q, and Y is a set of taskspecific labels. The NLSM task can be represented as estimating a conditional probability Pr (y|P,Q) based on the training set, and predicting the relationship for testing examples by y∗ = argmaxy∈Y Pr(y|P,Q). Concretely, for a paraphrase identification task, P and Q are two sentences, Y = {0, 1}, where y = 1 means that P and Q are paraphrase of each other, and y = 0 otherwise. For a natural language inference task, P is a premise sentence, Q is a hypothesis sentence, and Y = {entailment, contradiction, neutral} where entailment indicates Q can be inferred from P, contradiction indicates Q cannot be true condition on P, and neutral means P and Q are irrelevant to each other. In an answer sentence selection task, P is a question, Q is a candidate answer, and Y = {0, 1} where y = 1 means Q is a correct answer for P , and y = 0 otherwise.

## 3 Method

In this section, we first give a high-level overview of our model in Sub-section 3.1, and then give more details about our novel multi-perspective matching operation in Subsection 3.2.

## 3.1 Model Overview

We propose a bilateral multi-perspective matching (BiMPM) model to estimate the probability distribution Pr(y|P,Q). Our model belongs to the "matching-aggregation" framework [Wang and Jiang, 2016]. Contrarily to previous "matching-aggregation" approaches, our model matches P and Q in two directions (P → Q and P ← Q). In each individual direction, our model matches the two sentences from multiple perspectives. Figure 1 shows the architecture of our model. Given a pair of sentences P and Q, the BiMPMmodel estimates the probability distribution Pr(y|P,Q) through the following five layers.

Word Representation Layer. The goal of this layer is to represent each word in P and Q with a d-dimensional vector. We construct the d-dimensional vector with two components: a word embedding and a character-composed embedding. The word embedding is a fixed vector for each individual word, which is pre-trained with GloVe [Pennington et al., 2014] or word2vec [Mikolov et al., 2013]. The charactercomposed embedding is calculated by feeding each character (represented as a character embedding) within a word into a Long Short-Term Memory Network (LSTM) [Hochreiter and Schmidhuber, 1997], where the character embeddings are randomly initialized and learned jointly with other network parameters from NLSM tasks. The output of this layer are two sequences of word vectors P : [p1 , ..., pM ] and Q : [q1 , ..., qN ].

Figure 1: Architecture for Bilateral Multi-Perspective Matching (BiMPM) Model, where ⊗ is the multi-perspective matching operation described in sub-section 3.2.

![Image](https://ijcai.org/proceedings/2017/fulltext/579/image_000000_b0c5d1ed99cf1a93e872cfa09edc9cdacd34a2ccae269b5f75299ea4889bc288.png)

Context Representation Layer. The purpose of this layer is to incorporate contextual information into the representation of each time step of P and Q. We utilize a bi-directional LSTM (BiLSTM) to encode contextual embeddings for each time-step of P .

<!-- formula-not-decoded -->

Meanwhile, we apply the same BiLSTM to encode Q:

<!-- formula-not-decoded -->

Matching Layer. This is the core layer within our model. The goal of this layer is to compare each contextual embedding (time-step) of one sentence against all contextual embeddings (time-steps) of the other sentence. As shown in Figure 1, we will match the two sentences P and Q in two directions: match each time-step of P against all time-steps of Q, and match each time-step of Q against all time-steps of P . To match one time-step of a sentence against all time-steps of the other sentence, we design a multi-perspective matching operation ⊗. We will give more details about this operation in Sub-section 3.2. The output of this layer are two sequences of matching vectors (right above the operation ⊗ in Figure 1), where each matching vector corresponds to the matching result of one time-step against all time-steps of the other sentence.

Aggregation Layer. This layer is employed to aggregate the two sequences of matching vectors into a fixed-length matching vector. We utilize another BiLSTM model, and apply it to the two sequences of matching vectors individually. Then, we construct the fixed-length matching vector by concatenating (the four green) vectors from the last time-step of the BiLSTM models.

Prediction Layer. The purpose of this layer is to evaluate the probability distribution Pr(y|P,Q). To this end, we employ a two layer feed-forward neural network to consume the fixed-length matching vector, and apply the softmax function in the output layer. The number of nodes in the output layer is set based on each specific task described in Section 2.

## 3.2 Multi-perspective Matching Operation

We define the multi-perspective matching operation ⊗ in following two steps:

First, we define a multi-perspective cosine matching function fm to compare two vectors

<!-- formula-not-decoded -->

where v1 and v2 are two d-dimensional vectors, W ∈ ℜl×d is a trainable parameter with the shape l × d, l is the number of perspectives, and the returned value m is a l-dimensional vector m = [m1, ..., mk, ..., ml]. Each element mk ∈ m is a matching value from the k-th perspective, and it is calculated by the cosine similarity between two weighted vectors

<!-- formula-not-decoded -->

where ◦ is the element-wise multiplication, and Wk is the k-th row of W, which controls the k-th perspective and assigns different weights to different dimensions of the ddimensional space.

Second, based on fm, we define four matching strategies to compare each time-step of one sentence against all time-steps of the other sentence. To avoid repetition, we only define these matching strategies for one matching direction P → Q. The readers can infer equations for the reverse direction easily.

(1) Full-Matching. Figure 2 (a) shows the diagram of this matching strategy. In this strategy, each forward (or backward) contextual embedding -→h pi (or ←- h pi ) is compared with the last time step of the forward (or backward) representation of the other sentence -→h qN (or ←- h q1 ).

<!-- formula-not-decoded -->

(2) Maxpooling-Matching. Figure 2 (b) gives the diagram of this matching strategy. In this strategy, each forward (or backward) contextual embedding -→h pi (or ←- h pi ) is compared with every forward (or backward) contextual embeddings of the other sentence -→h qj (or ←- h qj ) for j ∈ (1...N), and only the maximum value of each dimension is retained.

Figure 2: Diagrams for different matching strategies, where fm is the multi-perspective cosine matching function in Eq.(3), the input includes one time step of one sentence (left orange block) and all the time-steps of the other sentence (right blue blocks), and the output is a vector of matching values (top green block) calculated by Eq.(3).

![Image](https://ijcai.org/proceedings/2017/fulltext/579/image_000001_a00274f421e2608dcd9c671fd6cf2e79bda8a6ac9fa331dbefa25436ebe32234.png)

<!-- formula-not-decoded -->

where max j∈(1...N)

<!-- formula-not-decoded -->

(3) Attentive-Matching. Figure 2 (c) shows the diagram of this matching strategy. We first calculate the cosine similarities between each forward (or backward) contextual embedding -→h pi (or ←- h pi ) and every forward (or backward) contextual embeddings of the other sentence -→h qj (or ←- h qj ):

<!-- formula-not-decoded -->

Then, we take -→α i,j (or ←- α i,j ) as the weight of -→h qj (or ←- h qj ), and calculate an attentive vector for the entire sentence Q by weighted summing all the contextual embeddings of Q:

<!-- formula-not-decoded -->

Finally, we match each forward (or backward) contextual embedding of -→h pi (or ←- h pi ) with its corresponding attentive vector:

<!-- formula-not-decoded -->

(4) Max-Attentive-Matching. Figure 2 (d) shows the diagram of this matching strategy. This strategy is similar to the Attentive-Matching strategy. However, instead of taking the weighed sum of all the contextual embeddings as the attentive vector, we pick the contextual embedding with the highest cosine similarity as the attentive vector. Then, we match each contextual embedding of the sentence P with its new attentive vector.

We apply all these four matching strategies to each timestep of the sentence P, and concatenate the generated eight vectors as the matching vector for each time-step of P. We also perform the same process for the reverse matching direction.

## 4 Experiments

In this section, we evaluate our model on three tasks: paraphrase identification, natural language inference and answer sentence selection. We will first introduce the general setting of our BiMPM models in Sub-section 4.1. Then, we demonstrate the properties of our model through some ablation studies in Sub-section 4.2. Finally, we compare our model with state-of-the-art models on some standard benchmark datasets in Sub-section 4.3, 4.4 and 4.5.

## 4.1 Experiment Settings

We initialize word embeddings in the word representation layer with the 300-dimensional GloVe word vectors pretrained from the 840B Common Crawl corpus [Pennington et al., 2014]. For the out-of-vocabulary (OOV) words, we initialize the word embeddings randomly. For the charactercomposed embeddings, we initialize each character as a 20-dimensional vector, and compose each word into a 50dimensional vector with a LSTM layer. We set the hidden size as 100 for all BiLSTM layers. We apply dropout to every layers in Figure 1, and set the dropout ratio as 0.1. To train the model, we minimize the cross entropy of the training set, and use the ADAM optimizer [Kingma and Ba, 2014] to update parameters. We set the learning rate as 0.001. During training, we do not update the pre-trained word embeddings. For all the experiments, we pick the model which works the best on the dev set, and then evaluate it on the test set.

## 4.2 Model Properties

To demonstrate the properties of our model, we choose the paraphrase identification task, and experiment on the "Quora Question Pairs" dataset 1 . This dataset consists of over

1 https://data.quora.com/First-Quora-Dataset-Release-QuestionPairs

Figure 3: Influence of the multi-perspective cosine matching function in Eq.(3) .

![Image](https://ijcai.org/proceedings/2017/fulltext/579/image_000002_fe695bb39a57433951a37fada9b7c9c33feaac8708e9e2caad7e22058c7b6193.png)

400,000 question pairs, and each question pair is annotated with a binary value indicating whether the two questions are paraphrase of each other. We randomly select 5,000 paraphrases and 5,000 non-paraphrases as the dev set, and sample another 5,000 paraphrases and 5,000 non-paraphrases as the test set. We keep the remaining instances as the training set 2 .

First, we study the influence of our multi-perspective cosine matching function in Eq.(3). We vary the number of perspectives l among {1, 5, 10, 15, 20}3 , and keep the other options unchanged. We also build a baseline model by replacing Eq.(3) with the vanilla cosine similarity function. Figure 3 shows the performance curve on the dev set, where l = 0 corresponds to the performance of our baseline model. We can see that, even if we only utilize one perspective (l = 1), our model gets a significant improvement. When increasing the number of perspectives, the performance improves significantly. Therefore, our multi-perspective cosine matching function is really effective for matching vectors.

Second, to check the effectiveness of bilateral matching, we build two ablation models to matching sentences in only a single direction: 1) "Only P → Q" which only matches P against Q; 2) "Only P ← Q" which only matches Q against P. Table 1 shows the performance on the dev set. Comparing the two ablation models with the "Full Model", we can observe that single direction matching hurts the performance for about 1 percent. Therefore, matching sentences in two directions is really necessary for acquiring better performance.

Third, we evaluate the effectiveness of different matching strategies. To this end, we construct four ablation models (w/o Full-Matching, w/o Maxpooling-Matching, w/o Attentive-Matching, w/o Max-Attentive-Matching) by eliminating a matching strategy at each time. Table 1 shows the performance on the dev set. We can see that eliminating any of the matching strategies would hurt the performance significantly.

2 We will release our source code and the dataset partition at https://zhiguowang.github.io/ .

3 Due to practical limitations, we did not experiment with more perspectives.

Table 1: Ablation studies on the dev set.

| Models                    |   Accuracy |
|---------------------------|------------|
| Only P → Q                |      87.74 |
| Only P ← Q                |      87.47 |
| w/o Full-Matching         |      87.86 |
| w/o Maxpooling-Matching   |      87.64 |
| w/o Attentive-Matching    |      87.87 |
| w/o MaxAttentive-Matching |      87.98 |
| Full Model                |      88.69 |

Table 2: Performance for paraphrase identification on the Quora dataset.

| Models                 |   Accuracy |
|------------------------|------------|
| Siamese-CNN            |      79.60 |
| Multi-Perspective-CNN  |      81.38 |
| Siamese-LSTM           |      82.58 |
| Multi-Perspective-LSTM |      83.21 |
| L.D.C.                 |      85.55 |
| BiMPM                  |      88.17 |

## 4.3 Experiments on Paraphrase Identification

In this Sub-section, we compare our model with state-of-theart models on the paraphrase identification task. We still experiment on the "Quora Question Pairs" dataset, and use the same dataset partition as Sub-section 4.2. This dataset is a brand-new dataset, and no previous results have been published yet. Therefore, we implemented three types of baseline models.

First, under the Siamese framework, we implement two baseline models: "Siamese-CNN" and "Siamese-LSTM". Both of the two models encode two input sentences into sentence vectors with a neural network encoder, and make a decision based on the cosine similarity between the two sentence vectors. But they implement the sentence encoder with a CNN and a LSTM respectively. We design the CNN and the LSTM model according to the architectures in [Wang et al., 2016c].

Second, based on the two baseline models, we implement two more baseline models "Multi-Perspective-CNN" and "Multi-Perspective-LSTM". In these two models, we change the cosine similarity calculation layer with our multiperspective cosine matching function in Eq.(3), and apply a fully-connected layer (with sigmoid function on the top) to make the prediction.

Third, we re-implement the "L.D.C." model proposed by [Wang et al., 2016d], which is a model under the "matchingaggregation" framework and acquires the state-of-the-art performance on several tasks.

Table 2 shows the performances of all baseline models and our "BiMPM" model. We can see that "Multi-PerspectiveCNN" (or "Multi-Perspective-LSTM") works much better than "Siamese-CNN" (or "Siamese-LSTM"), which further indicates that our multi-perspective cosine matching function (Eq.(3)) is very effective for matching vectors. Our "BiMPM" model outperforms the "L.D.C." model by more than two percent. Therefore, our model is very effective for the paraphrase identification task.

Table 3: Performance for natural language inference on the SNLI dataset.

| Models                         |   Accuracy |
|--------------------------------|------------|
| [Bowman et al., 2015]          |       77.6 |
| [Vendrov et al., 2015]         |       81.4 |
| [Mou et al., 2015]             |       82.1 |
| [Rockt¨aschel et al., 2015]    |       83.5 |
| [Liu et al., 2016b]            |       85.0 |
| [Liu et al., 2016a]            |       85.1 |
| [Wang and Jiang, 2015]         |       86.1 |
| [Cheng et al., 2016]           |       86.3 |
| [Parikh et al., 2016]          |       86.8 |
| [Munkhdalai and Yu, 2016]      |       87.3 |
| [Sha et al., 2016]             |       87.5 |
| [Chen et al., 2016] (Single)   |       87.7 |
| [Chen et al., 2016] (Ensemble) |       88.3 |
| Only P → Q                     |       85.6 |
| Only P ← Q                     |       86.3 |
| BiMPM                          |       86.9 |
| BiMPM (Ensemble)               |       88.8 |

## 4.4 Experiments on Natural Language Inference

In this Sub-section, we evaluate our model on the natural language inference task over the SNLI dataset [Bowman et al., 2015]. We test four variations of our model on this dataset, where "Only P → Q" and "Only P ← Q" are the single direction matching models described in Sub-section 4.2, "BiMPM" is our full model, and "BiMPM (Ensemble)" is an ensemble version of our "BiMPM" model. We design the ensemble model by simply averaging the probability distributions [Peng et al., 2015b; 2017] of four "BiMPM" models, and each of the "BiMPM" model has the same architecture, but is initialized with a different seed.

Table 3 shows the performances of the state-of-the-art models and our models. First, we can see that "Only P ← Q" works significantly better than "Only P → Q", which tells us that, for natural language inference, matching the hypothesis against the premise is more effective than the other way around. Second, our "BiMPM" model works much better than "Only P ← Q", which reveals that matching premise against the hypothesis can also bring some benefits. Finally, comparing our models with all the state-of-the-art models, we can observe that our single model "BiMPM" is on par with the state-of-the-art single models, and our 'BiMPM (Ensemble)" works much better than "[Chen et al., 2016] (Ensemble)". Therefore, our models achieve the state-of-the-art performance in both single and ensemble scenarios for the natural language inference task.

Table 4: Performance for answer sentence selection on TREC-QA and WikiQA datasets.

| Models                 | TREC-QA - MAP   | TREC-QA - MRR   | WikiQA - MAP   | WikiQA - MRR   |
|------------------------|-----------------|-----------------|----------------|----------------|
| [Yang et al., 2015]    | 0.695           | 0.763           | 0.652          | 0.665          |
| [Tan et al., 2015]     | 0.728           | 0.832           | -              | -              |
| Wang and Itty. [2015]  | 0.746           | 0.820           | -              | -              |
| [Santos et al., 2016]  | 0.753           | 0.851           | 0.689          | 0.696          |
| [Yin et al., 2015]     | -               | -               | 0.692          | 0.711          |
| [Miao et al., 2016]    | -               | -               | 0.689          | 0.707          |
| [Wang et al., 2016d]   | 0.771           | 0.845           | 0.706          | 0.723          |
| [He and Lin, 2016]     | 0.777           | 0.836           | 0.709          | 0.723          |
| [Rao et al., 2016]     | 0.801           | 0.877           | 0.701          | 0.718          |
| [Wang et al., 2016a]   | -               | -               | 0.734          | 0.742          |
| [Wang and Jiang, 2016] | -               | -               | 0.743          | 0.755          |
| BiMPM                  | 0.802           | 0.875           | 0.718          | 0.731          |

## 4.5 Experiments on Answer Sentence Selection

In this Sub-section, we study the effectiveness of our model for answer sentence selection tasks. The answer sentence selection task is to rank a list of candidate answer sentences based on their similarities to the question, and the performance is measured by the mean average precision (MAP) and mean reciprocal rank (MRR). We experiment on two datasets: TREC-QA [Wang et al., 2007] and WikiQA [Yang et al., 2015]. Experimental results of the state-of-the-art models 4 and our "BiMPM" model are listed in Table 4, where the performances are evaluated with the standard trec eval-8.0 script 5 . We can see that the performance from our model is on par with the state-of-the-art models. Therefore, our model is also effective for answer sentence selection tasks.

## 5 RelatedWork

Natural language sentence matching (NLSM) has been studied for many years. Early approaches focused on designing hand-craft features to capture n-gram overlapping, word reordering and syntactic alignments phenomena [Heilman and Smith, 2010; Wang and Ittycheriah, 2015]. This kind of method can work well on a specific task or dataset, but it's hard to generalize well to other tasks.

With the availability of large-scale annotated datasets [Bowman et al., 2015], many deep learning models were proposed for NLSM. The first kind of framework is based the Siamese architecture [Bromley et al., 1993], where sentences are encoded into sentence vectors based on some neural network encoders, and then the relationship between two sentences was decided solely based on the two sentence vectors [Bowman et al., 2015; Yang et al., 2015; Tan et al., 2015]. However, this kind of framework ignores the fact that the lower level interactive features between two sentences are indispensable. Therefore, many neural network models were proposed to match sentences from multiple level of granularity [Yin et al., 2015; Wang and Jiang, 2016; Wang et al., 2016d]. Experimental results on many tasks have proofed that the new framework works significantly better than the previous methods. Our model also belongs to this framework, and we have shown its effectiveness in Section 4.

4 [Rao et al., 2016] pointed out that there are two versions of TREC-QA dataset: raw-version and clean-version. In this work, we utilized the clean-version. Therefore, we only compare with approaches reporting performance on this dataset.

5 http://trec.nist.gove/trec eval/

## 6 Conclusion

In this work, we propose a bilateral multi-perspective matching (BiMPM) model under the "matching-aggregation" framework. Different from the previous "matchingaggregation" approaches, our model matches sentences P and Q in two directions (P → Q and P ← Q). And, in each individual direction, our model matches the two sentences from multiple perspectives. We evaluated our model on three tasks: paraphrase identification, natural language inference and answer sentence selection. Experimental results on standard benchmark datasets show that our model achieves the state-of-the-art performance on all tasks.

## References

1. [Bowman et al., 2015] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.
2. [Bromley et al., 1993] Jane Bromley, James W. Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. IJPRAI, 7(4):669- 688, 1993.
3. [Chen et al., 2016] Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang. Enhancing and combining sequen- tial and tree lstm for natural language inference. arXiv preprint arXiv:1609.06038, 2016.
4. [Cheng et al., 2016] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733, 2016.
5. [He and Lin, 2016] Hua He and Jimmy Lin. Pairwise word interaction modeling with deep neural networks for se- mantic similarity measurement. In NAACL, 2016.
6. [Heilman and Smith, 2010] Michael Heilman and Noah A Smith. Tree edit models for recognizing textual entail- ments, paraphrases, and answers to questions. In NAACL, 2010.
7. [Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
8. [Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
9. [LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521(7553):436- 444, 2015.
10. [Liu et al., 2016a] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Modelling interaction of sentence pair with coupled-lstms. arXiv preprint arXiv:1605.05573, 2016.
11. [Liu et al., 2016b] Yang Liu, Chengjie Sun, Lei Lin, and Xi- aolong Wang. Learning natural language inference us- ing bidirectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090, 2016.
12. [Miao et al., 2016] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In ICML, 2016.
13. [Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119, 2013.
14. [Mou et al., 2015] Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. Natural language in- ference by tree-based convolution and heuristic matching. arXiv preprint arXiv:1512.08422, 2015.
15. [Munkhdalai and Yu, 2016] Tsendsuren Munkhdalai and Hong Yu. Neural tree indexers for text understanding. arXiv preprint arXiv:1607.04492, 2016.
16. [Parikh et al., 2016] Ankur P Parikh, Oscar Täckström, Di- panjan Das, and Jakob Uszkoreit. A decomposable atten- tion model for natural language inference. arXiv preprint arXiv:1606.01933, 2016.
17. [Peng et al., 2015a] Xi Peng, Junzhou Huang, Qiong Hu, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. From circle to 3-sphere: Head pose estimation by instance parameterization. Computer Vision and Image Understanding, 136:92-102, 2015.
18. [Peng et al., 2015b] Xi Peng, Shaoting Zhang, Yu Yang, and Dimitris N Metaxas. Piefa: Personalized incremental and ensemble face alignment. In Proceedings of the IEEE In- ternational Conference on Computer Vision, pages 3880- 3888, 2015.
19. [Peng et al., 2016] Xi Peng, Rogerio S Feris, Xiaoyu Wang, and Dimitris N Metaxas. A recurrent encoder-decoder net- work for sequential face alignment. In European Confer- ence on Computer Vision, pages 38-56. Springer Interna- tional Publishing, 2016.
20. [Peng et al., 2017] Xi Peng, Shaoting Zhang, Yang Yu, and Dimitris N Metaxas. Toward personalized modeling: In- cremental and ensemble alignment for sequential faces in the wild. International Journal of Computer Vision, pages 1-14, 2017.
21. [Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
22. [Rao et al., 2016] Jinfeng Rao, Hua He, and Jimmy Lin. Noise-contrastive estimation for answer selection with deep neural networks. In CIKM, 2016.
23. [Rocktäschel et al., 2015] Tim Rocktäschel, Edward Grefen- stette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. Reasoning about entailment with neural atten- tion. arXiv preprint arXiv:1509.06664, 2015.
24. [Santos et al., 2016] Cicero dos Santos, Ming Tan, Bing Xi- ang, and Bowen Zhou. Attentive pooling networks. arXiv preprint arXiv:1602.03609, 2016.
25. [Sha et al., 2016] Lei Sha, Baobao Chang, Zhifang Sui, and Sujian Li. Reading and thinking: Re-read lstm unit for textual entailment recognition. In COLING, 2016.
26. [Tan et al., 2015] Ming Tan, Cicero dos Santos, Bing Xi- ang, and Bowen Zhou. Lstm-based deep learning mod- els for non-factoid answer selection. arXiv preprint arXiv:1511.04108, 2015.
27. [Vendrov et al., 2015] Ivan Vendrov, Ryan Kiros, Sanja Fi- dler, and Raquel Urtasun. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361, 2015.
28. [Wang and Ittycheriah, 2015] Zhiguo Wang and Abraham It- tycheriah. Faq-based question answering via word align- ment. arXiv preprint arXiv:1507.02628, 2015.
29. [Wang and Jiang, 2015] Shuohang Wang and Jing Jiang. Learning natural language inference with lstm. arXiv preprint arXiv:1512.08849, 2015.
30. [Wang and Jiang, 2016] Shuohang Wang and Jing Jiang. A compare-aggregate model for matching text sequences. arXiv preprint arXiv:1611.01747, 2016.
31. [Wang et al., 2007] Mengqiu Wang, Noah A Smith, and Teruko Mitamura. What is the jeopardy model? a quasi- synchronous grammar for qa. In EMNLP, 2007.
32. [Wang et al., 2016a] Bingning Wang, Kang Liu, and Jun Zhao. Inner attention based recurrent neural networks for answer selection. In ACL, 2016.
33. [Wang et al., 2016b] Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. Multi-perspective context match- ing for machine comprehension. arXiv preprint arXiv:1612.04211, 2016.
34. [Wang et al., 2016c] Zhiguo Wang, Haitao Mi, and Abraham Ittycheriah. Semi-supervised clustering for short text via deep representation learning. In CoNLL, 2016.
35. [Wang et al., 2016d] Zhiguo Wang, Haitao Mi, and Abra- ham Ittycheriah. Sentence similarity learning by lexical decomposition and composition. In COLING, 2016.
36. [Yang et al., 2015] Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain ques- tion answering. In EMNLP, 2015.
37. [Yin et al., 2015] Wenpeng Yin, Hinrich Schütze, Bing Xi- ang, and Bowen Zhou. Abcnn: Attention-based convolu- tional neural network for modeling sentence pairs. arXiv preprint arXiv:1512.05193, 2015.