Dual-View Variational Autoencoders for Semi-Supervised Text Matching

1 Introduction

The need to semantically match two text sequences arises in many Natural Language Processing problems, where a central task is to compute the matching degree between two sentences and determine their semantic relationship. For instance, in Paraphrase Identification [Dolan and Brockett, 2005], whether one sentence is a paraphrase of another has to be determined; In Question Answering [Yang et al., 2015], a matching score is calculated for a question and each of its candidate answers for making decisions; And in Natural Language Inference [Bowman et al., 2015], the relationship between a premise and a hypothesis is classified as entailment, neutral or contradiction.

Most previous studies on text matching focus on developing supervised models with deep neural networks. These models can be essentially divided into two categories: (i) sentence encoding-based models, which separately encode each of the two sentences into a vector representation (sentence embedding) and then match between the two vectors [Bowman et al., 2016a; Mueller and Thyagarajan, 2016], and (ii)

(b) sentence pair interaction model

Figure 1: Two categories of text matching models: They leverage sentence embeddings and word-level interaction features, respectively, which can be seen as two distinct views of the sentence pair.

sentence pair interaction models, which use some sorts of word alignment methods, such as interaction matrices [Gong et al., 2018; Wu et al., 2018] or attention mechanisms [Rockt¨aschel et al., 2016; Wang and Jiang, 2017], to obtain finegrained interaction features for predicting the matching degree. Sentence encoding-based models leverage global sentence representations with high-level semantic features, while sentence pair interaction models leverage word-by-word interaction features containing local matching patterns, as illustrated in Figure 1.

With the recent advances in deep generative models, some studies begin to employ variational autoencoders (VAEs) [Kingma and Welling, 2014] to learn informative sentence embeddings for various downstream NLP problems, including text matching [Bowman et al., 2016b; Zhao et al., 2018; Shen et al., 2018]. They leverage a VAE to encode sentences into latent codes, which are used as sentence embeddings for a sentence encoding-based matching model. The VAE and the matching model can be jointly trained in a semisupervised manner, leveraging large amounts of unlabeled data to improve matching performance. However, these models are limited to global semantic features in the sentence embeddings, leaving out the word-level interaction features that have been proved important for predicting matching degrees in the supervised case [Lan and Xu, 2018].

Motivated by these observations, we propose to unify the sentence-level embedding features and the word-level interaction features within a variational autoencoder, leveraging both labeled and unlabeled sentence pairs in a semisupervised manner for text matching. We take inspiration from Co-Training [Blum and Mitchell, 1998], where two classifiers over two distinct views of the data examples are trained to produce consistent predictions on the unlabeled data. For a sentence pair, the aforementioned two levels of features are taken as two distinct views, namely the embedding view and the interaction view. The proposed model is denoted Dual-View Variational AutoEncoder (DV-VAE) (Figure 2). In the generative process, two sentences are generated from two latent variables, respectively. The matching degree is also generated from these two latent variables, treated as the embedding view, through a sentence encoding-based model. In the inference process, the matching degree is inferred from the interaction view through a sentence pair interaction model. During the optimization of the variational lower bound, the two matching models implicitly provide pseudo labels on unlabeled data for each other, which can be interpreted as an implicit Co-Training mechanism.

Our contributions are as follows: (i) We propose DualView Variational AutoEncoder (DV-VAE) to unify the embedding view and the interaction view of a sentence pair for semi-supervised text matching. An implicit Co-Training mechanism is also formulated to interpret the training process. (ii) We instantiate an implementation of DV-VAE and adopt a novel sentence pair interaction matching model, where interaction matrices across words and contexts are introduced to enrich the interaction features. (iii) Using three datasets: SNLI, Quora and a Community QA dataset, we empirically demonstrate the superiority of DV-VAE over several strong semi-supervised and supervised baselines.

2 Dual-ViewVariational Antoencoder

Suppose that we have a labeled sentence pair set Dl and an unlabeled sentence pair set Du. (x1, x2, y) ∈ Dl denotes a labeled sentence pair, where x1, x2 are two sentences and y ∈ {1, 2, . . . , C} is the matching degree of x1 and x2. Here y is discretized and text matching is treated as a classification problem. Similarly, (x1, x2) ∈ Du denotes an unlabeled pair. Our goal is to develop a semi-supervised text matching model using both the labeled and unlabeled data Dl and Du, which can improve upon the performance of supervised text matching models using the labeled data Dl only.

2.1 Model Architecture

The probabilistic graphical model of DV-VAE is shown in Figure 2. It consists of a generative model matching from the embedding view and an inference model matching from the interaction view.

Generative Model. The generative process of a sentence pair and their matching degree (x1, x2, y) is defined as follows: two continuous latent codes z1, z2 ∈ RdZ are independently sampled from a prior p(z), and are used to generate x1 and x2 through a decoder pθ(x|z). Latent codes z1, z2 are also fed into a sentence encoding-based matching model pθ(y|z1, z2) to generate the matching degree y. The joint distribution can be explained by the following factorization:

where p(z1) = p(z2) = p(z) = N(z; 0, I) is a Gaussian prior. And pθ(y|z1, z2) is referred to as the embedding matcher as it matches from the embedding view (latent space).

Inference Model. According to the conditional independence properties in the generative model, the variational posterior qφ(z1, z2, y|x1, x2) can be factorized as:

where we model qφ(y|z1, z2) by the embedding matcher pθ(y|z1, z2). qφ(z1, z2, y|x1, x2) can also be factorized as:

where we model qφ(y|x1, x2) by a sentence pair interaction matching model to match from the interaction view. Thus qφ(y|x1, x2) is referred to as the interaction matcher and is adopted to make predictions at test time. In analogy to CoTraining [Blum and Mitchell, 1998], we assume that each of the embedding view and the interaction view is sufficient to train the corresponding matcher, and the predictions from the two matchers are consistent in the inference process: qφ(y|x1, x2) = pθ(y|z1, z2). With this consistency assumption, we obtain the following from Equ (1) and Equ (2):

which are taken as the inference model in labeled and unlabeled cases, respectively. Here encoders qφ(z1|x1) = N(z1; µφ(x1), diag(σ2φ (x1))) and qφ(z2|x2) = N(z2; µφ(x2), diag(σ2φ (x2))) are diagonal Gaussians.

Objective

The variational lower bound of the data likelihood is used as the objective, for both the labeled and unlabeled data.

For a labeled sentence pair (x1, x2, y),

We rewrite L(x1, x2, y) as

where -R(x1, x2) = Eqφ(z1|x1)qφ(z2|x2)[- log pθ(x1|z1) - log pθ(x2|z2)] is the reconstruction loss of x1 and x2; -D(x1, x2, y) = Eqφ(z1|x1)qφ(z2|x2)[- log pθ(y|z1, z2)] can be seen as an expected discriminative loss for the embedding matcher pθ(y|z1, z2); and the last two KL-divergence terms regularize the posteriors to be close to the priors.

For an unlabeled sentence pair (x1, x2),

log pθ(x1, x2)

Since qφ(y|x1, x2) is not included in the expression of L(x1, x2, y), we explicitly add a discriminative loss for qφ(y|x1, x2), weighted by α:

Finally, we obtain the objective function to be minimized on the entire dataset Dl ∪ Du:

2.2 Implicit Co-Training

In this section, we show that the training process of DVVAE is implicitly related to Co-Training [Blum and Mitchell, 1998], where two classifiers are iteratively trained to explicitly provide pseudo labels on unlabeled data for each other. Since in DV-VAE the embedding matcher pθ(y|z1, z2) and the interaction matcher qφ(y|x1, x2) are simultaneously trained through the optimization of J in Equ (6), we analyze their gradients to study the training process. For clarity, we specify the parameters in qφ(y|x1, x2) as φm and the parameters in pθ(y|z1, z2) as θm, respectively.

(i) For a labeled sentence pair, minimizing Lα(x1, x2, y)

also minimizes the discriminative losses (-D(x1, x2, y) and - log qφ(y|x1, x2)) for the two matchers, which are independently trained just as in supervised learning.

(ii) For an unlabeled sentence pair, we analyze the gradi-

ents of U(x1, x2) in Equ (4) w.r.t. θm and φm, respectively.

For the embedding matcher pθm (y|z1, z2),

where the discriminative gradient ∇θm[-D(x1, x2, y)] is reweighted by the predicted distribution qφm (y|x1, x2) from the interaction matcher.

For the interaction matcher qφm (y|x1, x2), 1

The first term can be seen as an application of the REINFORCE algorithm [Williams, 1992] from Reinforcement Learning, such that D(x1, x2, y), matching degree y, sentence pair (x1, x2) and qφm (y|x1, x2) correspond to the reward signal, action, state and decision model, respectively [Mnih and Gregor, 2014; Xu et al., 2017]. The second term maximizes the entropy of qφm (y|x1, x2), and is treated as a regularizer.

With the training process going on, the two matchers distinguish correct and incorrect labels better and better through the supervised loss Lα(x1, x2, y). Therefore, for unlabeled sentence pairs, the weight for the embedding matcher's discriminative gradient in Equ (7) becomes larger on correct ys, and the interaction matcher receives larger reward signals when it gives correct predictions in Equ (10). This is an alternative way of providing pseudo labels for unlabeled data, and can be treated as an implicit Co-Training mechanism.

2.3 Model Implementation

We present an implementation of DV-VAE (shown in Figure 3) in detail, which consists of an encoder qφ(z|x), a decoder pθ(x|z), an embedding matcher pθ(y|z1, z2) and an interaction matcher qφ(y|x1, x2).

1 To derive Equ (8), note that φm does not exist in L(x1, x2, y); to derive Equ (9), note that ∑y K∇ωq(y; ω) = K∇ω ∑y q(y; ω) = K∇ω1 = 0 when K is irrelevant with y, which is the case for R(x1, x2) and the KL terms in L(x1, x2, y); to derive Equ (10), use ∇ωq(y; ω) = q(y; ω)∇ω log q(y; ω).

Encoder qφ(z|x). We adopt a bidirectional LSTM (BiLSTM) as the encoder. The last hidden states of the forward and backward directions are concatenated and fed into two Multi-Layer Perceptrons (MLPs) to compute the mean µ and the standard deviation σ for qφ(z|x) = N(z; µ, diag(σ2 )).

Decoder pθ(x|z). To avoid the training collapse of VAEbased text generation models, we adopt a dilated CNN sequence decoder that is most similar to the one in [Yang et al., 2017]: Latent code z is concatenated to every word embedding of x to serve as the decoder input. Every feature dimension of the decoder input is treated as a channel and masked one-dimensional convolution proceeds in the time dimension. Dilated convolution with rate r is applied so that one in every r consecutive inputs is picked to convolve with the filter. Multiple dilated convolution layers are stacked with exponentially increasing dilation rates [20 , 21 , 22 , . . . ], and every layer is wrapped in a bottleneck residual block [He et al., 2016] to ease optimization. The outputs of the last layer are fed into a fully connected layer followed by a softmax nonlinearity to produce the probability pθ(wj|w<j, z) for all time steps j ∈ [1, . . . , T]. Then the reconstruction probability of text sequence x is computed as pθ(x|z) = ∏Tj=1 pθ(wj |w<j , z).

Embedding matcher pθ(y|z1, z2). We adopt an MLP taking as input the concatenation of z1, z2, the element-wise difference z1 - z2 and the element-wise product z1 ⊙ z2:

Interaction matcher qφ(y|x1, x2). For x1 = {w11,w12, . . . ,w1T1 }, we denote the word embedding sequence as E1 = {e11, e12, . . . , e1T1 } ,where e1j ∈ RdE is the word embedding for token w1j. Then a bidirectional LSTM (Bi-LSTM) is adopted to get a context sequence H1 = {h11, h12, . . . , h1T1 }, where h1j = [-→ h1j ; ←- h1j] is the concatenation of the corresponding forward and backward hidden states of the Bi-LSTM, and -→ h1j , ←- h1j ∈ RdH . Similarly, we have E2 = {e21, e22, . . . , e2T2 } and H2 = {h21, h22, . . . , h2T2 } for x2.

We match every word embedding in E1 with those in E2, and match every context in H1 with those in H2. We also cross-match the contexts in H1 (or H2) with the words in E2 (or E1) to catch the matching patterns between contexts and words. Therefore, we obtain four interaction matrices M1,M2,M3,M4 ∈ RT1×T2 :

where we set dE = dH to allow for the dot product between -→ hij (or ←- hij) and eij. Then M1,M2,M3,M4 are stacked as a four-channel input for a CNN, whose output is passed through an MLP to predict the final matching degree y.

3 Experiments

Using three datasets, we show the superiority of DV-VAE over strong semi-supervised and supervised baselines.

3.1 Experimental Setup

Datasets. We experiment on three datasets: SNLI [Bowman et al., 2015] for Natural Language Inference, Quora Question Pairs for Paraphrase Identification, and a Community Question Answering (CQA) dataset [Nakov et al., 2015] for Question Answering. Statistics of these datasets are summarized in Table 2. (i) We perform simulated semisupervised experiments with different amounts of labeled data along the same line as previous studies [Shen et al., 2018; Zhao et al., 2018]: for SNLI, we select 5.25%, 10.8% and 22.2% of the original train set to be Dl (i.e., approximately 28k, 59k and 120k labeled pairs), and remove the labels of the remaining data in the train set to make up Du; for Quora, we select 1k, 5k, 10k and 25k labeled pairs in the train set. We experiment on five random labeled/unlabeled splits of the train set for each amount of labeled data, and report the mean and standard deviation of the matching accuracies. (ii) For the CQA dataset, the original train set is used as Dl, and we additionally adopt WikiQA [Yang et al., 2015], which has 29k QA pairs, as Du by removing all its labels. 2

Model Configurations. We set dE = 300 and dZ = 500. Word embeddings are initialized with Glove [Pennington et al., 2014]. We share the parameters of Bi-LSTMs in the encoder and the interaction matcher. Hidden state size dH is set to 300 for both directions. For the decoder, we choose a 3-layer dilated CNN, with dilation rates [1, 2, 4]. In all the bottleneck residual blocks, filter size is set to 3 and channel numbers are set to 300 internally and 600 externally. In the interaction matcher, we adopt a 2-layer CNN with filter sizes 5×5×8 and 3×3×16 such that a dynamic pooling is after the first layer to get 4×4 fixed-sized feature maps and a max pooling is after the second layer, followed by a 3-layer MLP with 16, 8 and C hidden units. ReLU is used as the nonlinearity and Batch Normalization is adopted in each layer.

Training Details. We use the reparameterization trick and sample one z from qφ(z|x) to estimate the variational lower bounds [Kingma and Welling, 2014]. α in Equ (5) is set to 20. We substitute the two KL-divergence terms in L(x1, x2, y) with max(γ, KL(qφ(z1|x1)‖p(z1)) + KL(qφ(z2|x2)‖p(z2))) to force the decoder rely more on latent codes [Yang et al., 2017]. We set γ = 10 for SNLI, and γ = 20 for the other experiments. SGD with momentum 0.9 and weight decay 1×10-3 is adopted in optimization. We use an initial learning rate of 3 × 10-3 . Batch size is tuned on {32, 64, 128} for each experiment. We sample half of the minibatch from Dl and half from Du in each iteration. We adopt early stopping where performance on dev set is evaluated every time Dl is traversed. A dropout rate of 0.1 is used in each layer of the decoder net. Experiments are implemented in PyTorch.

2 We use the train/dev/test split of [Wang et al., 2017] on Quora. Due to memory limitations, we truncate the texts in Quora, CQA and WikiQA to have no more than 100, 500 and 100 tokens, respectively.

3.2 Evaluations on Text Matching

Natural Language Inference. First, we compare DV-VAE with semi-supervised baselines that combine autoencoders with sentence-encoding based matching models, and the results are reported in Table 1. Results indicate that DV-VAE consistently outperforms all the semi-supervised baselines by a large margin (3.9% ∼ 6.5%) under all the 3 labeled data sizes. These results demonstrate the importance of incorporating the interaction view in DV-VAE for semi-supervised text matching. Second, we report in Table 1 the results from our interaction matcher trained on Dl in a supervised manner. DV-VAE consistently outperforms the supervised interaction matcher, verifying its effectiveness on using unlabeled data to improve supervised learning.

Paraphrase Identification. We get similar results on Quora, as shown in Table 1. DV-VAE's accuracy gains over the semi-supervised baselines are consistently more than 3% for all the 4 labeled data sizes. DV-VAE also achieves further accuracy gains over the supervised interaction matcher, and when labeled data is scarce (|Dl| = 1k), the absolute improvement is up to 4.4%.

Community Question Answering. We compare our model with several strong supervised baselines in Table 3. These baselines and our interaction matcher are trained on Dl and DV-VAE is trained on Dl and Du (WikiQA). Results show that DV-VAE outperforms all the baselines by leveraging the additional 29k unlabeled WikiQA sentence pairs, achieving an accuracy gain of 1.3% over the state of the art method KEHNN [Wu et al., 2018]. Note that KEHNN leverages additional prior knowledge of the QA pairs while we leverage additional unlabeled QA pairs. This indicates that sufficient amount of unlabeled data may play the role of prior knowledge in terms of the performance improvement.

3.3 Model Visualization

We first visualize the reward D(x1, x2, y) for the interaction matcher in Figure 4 to justify the implicit CoTraining mechanism. The distributions of D(x1, x2, ycorrect) and D(x1, x2, yincorrect) on Du are distinguishable, and the mean for D(x1, x2, ycorrect) is larger than that for D(x1, x2, yincorrect), which is statistically significant (p < 0.01). This demonstrates that the interaction matcher may receive larger reward signals by predicting correct ys than incorrect ones on the unlabeled data. With the embedding matcher providing useful reward signals, the interaction matcher can effectively leverage unlabeled data.

We then visualize the learned decoder and embedding matcher in DV-VAE by generating labeled sentence pairs (x1, x2, y) from latent codes z1, z2 sampled from p(z) = N(z; 0, I). Some generated examples are shown in Table 4, demonstrating the capability of DV-VAE to learn the data manifold that is useful for semi-supervised classification.

4 RelatedWork

Learning to match text sequences is a long standing problem and most state of the art methods use a compare-aggregate architecture [Wang and Jiang, 2017], such as DIIN [Gong et al., 2018], CSRAN [Tay et al., 2018], MwAN [Tan et al., 2018] and KEHNN [Wu et al., 2018]. [Lan and Xu, 2018] compared a broad range of text matching models over eight datasets. They all focus on supervised learning while we explore semi-supervised methods leveraging unlabeled text pairs to improve the performance of supervised methods.

Close to our work are recent applications of variational autoencoders [Kingma and Welling, 2014] and NVIL [Mnih and Gregor, 2014] in NLP. (i) Some focused on modeling a single piece of text: [Mnih and Gregor, 2014] and [Miao et al., 2016] used bag of words methods for document modeling; [Bowman et al., 2016b; Yang et al., 2017] and others explored VAE and various improved models to generate

• x1: a child is playing with a soccer ball in the grass .
• x2: a man is getting ready to throw a bowling ball down the lane .
• y: neutral
• x1: the woman is sitting on the bench .
• x2: a woman in a black shirt is walking down the street .
• y: contradiction
• x1: a group of people are riding on a roller coaster .
• x2: there is a group of people playing soccer .
• y: contradiction
• x1: two dogs are playing with a red ball .
• x2: the dog is running .
• y: entailment
• x1: two young men , one wearing a white shirt and the other wearing a white shirt , are sitting on a bench .
• x2: there is a man in a white shirt .
• y: entailment

Table 4: Sentence pairs generated by DV-VAE trained on SNLI (|Dl| = 28k).

natural language sentences; [Xu et al., 2017] adopted semisupervised VAE proposed in [Kingma et al., 2014] for text classification. (ii) Others developed specific VAE structures for sequence transduction tasks such as sentence compression [Miao and Blunsom, 2016], machine translation [Zhang et al., 2016], dialogue generation [Serban et al., 2017]. (iii) To our knowledge, there are few studies modeling a pair of texts with VAE except [Shen et al., 2018] that adopted deconvolutional networks in a VAE for semi-supervised text matching. However, this method matches texts from the embedding view only, while we further combine the interaction view.

Our work is also related to Multi-View Learning [Xu et al., 2013] where features can be separated into distinct subsets (views). Particularly relevant are Co-Training [Blum and Mitchell, 1998] and Co-Regularization [Sindhwani et al., 2005], where two models train each other on unlabeled data. However, instead of explicitly designing an algorithm or an objective to enable the Co-Training mechanism, we implicitly achieve it by maximizing the variational lower bound.

5 Conclusions

In this study, we have proposed Dual-View Variational AutoEncoder (DV-VAE) to unify the embedding view and the interaction view for semi-supervised text matching. Gradient analysis has also revealed an implicit Co-Training mechanism to explain the semi-supervised learning process. Finally, our experimental study has verified the effectiveness of DVVAE. Further, our work is a step towards combining multiview learning with neural network models, which seems a promising strategy for semi-supervised deep learning.

Acknowledgments

This work is supported in part by National Key R&D Program of China 2018YFB1700403, NSFC U1636210&61421003, Shenzhen Institute of Computing Sciences, and the Fundamental Research Funds for the Central Universities.