A Variational Autoencoding Approach for Inducing Cross-lingual Word Embeddings

1 Introduction

Distributed representations have become an increasingly important tool in machine learning. Typically, word embeddings are trained to represent words in a continuous space in an unsupervised way, which characterizes the lexicosemantic relations among words. In many NLP tasks, they prove to be high-quality features, in contrast to hand-crafted, and thus expensive features. Successful applications of distributed representations include sentence modeling[Bengio et al., 2003], sentiment analysis[Socher et al., 2011] and document classification[Klementiev et al., 2012].

The use of distributed representations is motivated by the idea that they capture semantics and syntax as well as encoding similarity between words. Like words have synonyms in the same language, word pairs across language also share resembling semantics. Mikolov[Mikolov et al., 2013] observed a strong similarity between vector spaces across languages and suggested a linear cross-lingual mapping between the two vector spaces is technically possible.

Recently, it is becoming more and more desirable to have unsupervised techniques which can learn useful syntactic and semantic features that are invariant to tasks or languages which we are interested in. With accurate joint-space embeddings of both language, one can develop abundant textual resources from language A to build models for language B. This is especially useful for transferring limited label information from high-resources to low-resources languages, and has been demonstrated effective for document classification by Klementiev[Klementiev et al., 2012], whose model outperforms a strong phrase based machine translation baseline.

⇤Corresponding Author

Defining a joint space objective function is crucial at the core of cross-lingual learning. Several models for training cross-lingual embeddings have been proposed, usually starting from a monolingual objective following cross-lingual objective as constraints. [Zou et al., 2013] learned word embeddings of different languages in separate spaces with monolingual corpus and projected the embeddings into a joint space. [Mikolov et al., 2013] learned a linear projection from one embedding space to another. It was extended by [Faruqui and Dyer, 2014], who simultanteously projected source and target language embeddings into a joint space using canonical correlation analysis. Bilingual autoencoder for bags-of-words (BAE)[AP et al., 2014] reconstructed the bag-of-words representation of semantic equivalent sentence pairs. BilBOWA[Gouws et al., ] extended CBOW and skipgram models with minimizing differences between parallel sentence pair representations. [ˇSuster et al., 2016] learned multi sense word embeddings with discrete autoencoders. Many of these models can be viewed as instances of a more general framework for inducing cross-lingual word embeddings, which integrates monolingual objectives(similar words in each language have similar embeddings) and cross-lingual objectives(similar words across languages have similar representations).

Inspired by the recent advances[Kingma and Welling, 2013; Rezende et al., 2014] in neural variational inference, we propose a variational autoencoding model(BiVAE) to cross-lingual learning. Unlike the framework mentioned above, we explicitly model the underlying semantics of bilingual sentence pairs(see Figure1). Similar to [Suzuki et al., 2016], we make two assumptions about BiVAE:

• ÷ There exists a continuous latent variable z from this underlying semantic space.
• ÷ This variable z, guides the generative process of the equivalent sentence pairs x and y independently, i.e.

With this assumption, the following formulation characterizes our probabilistic language model:

This brings the benefits that the latent variable z serves as a global variable capturing underlying semantics between parallel sentence pairs, thus forcing the words from different language to be embedded in a unified vector space without the help of extra alignment constraints. However the incorporation of it into the probabilistic language model brings one challenge: the posterior inference in this model is intractable. In order to address this issue, we propose the variational encoder-decoder model to cross-lingual learning, motivated by recent success of variational neural inference[Kingma and Welling, 2013; Rezende et al., 2014]. We employ deep neural networks to model the posterior distribution of the latent hidden variable as they are capable of learning highly nonlinear functions thus making the inference tractable. In order to train model parameters efficiently, we apply a reparameterization trick[Kingma and Welling, 2013; Rezende et al., 2014] on the variational lower bound which enables us using stochastic gradient optimization during training. Specifically, BiVAE has two essential components:

• ÷ A variational inference network infers the posterior distribution of z according to the encoded representation of parallel sentence pairs(i.e. qφ(z ♣ x↪ y)).

Model details will be introduced in section 3. We train the cross-lingual word embeddings with the proposed model and apply these embeddings to a standard document classification task and show that training with parallel corpus only, our model performs comparably with previous reported state-ofthe-art models.

• ÷ A decoder network reconstructs the observable sentences conditioned on the inferred distribution of z(i.e. p✓(x ♣ z), p✓(y ♣ z)).

2 Background: Variational Autoencoder

In this section, we briefly review the VAE[Kingma and Welling, 2013; Rezende et al., 2014]. The VAE is a generative model which is based on a regularized version of the standard autoencoder. It modifies the autoencoder architecture by replacing the deterministic function φenc with a learnable posterior recognition model, q(z ♣ x). The VAE imposes a prior distribution on the hidden variable z which enforces a regular geometry over the hidden representation and makes it possible to draw proper samples from the model using ancestral sampling. Intuitively, the VAE learns codes not as single points, but as soft ellipsoidal regions in latent space, forcing the codes to fill the space rather than memorizing the training data as isolated codes.

Given an observed variable x, the VAE introduces a latent variable z, and assumes that x is generated from z which can be characterized by the following formula:

where ✓ denotes the generative parameter of the model and p(z) denotes the prior distribution of the latent variable z, e.g Gaussian distribution. p✓(x ♣ z) is the conditional distribution that models the generative process of x conditioned on the hidden variable z, typically estimated via a non-linear deep neural network.

The integration of z brings a challenge on the posterior inference and VAE adopts two techniques to tackle this problem: variational neural inference and reparameterization.

Variational neural inference employs deep neural network to approximate the posterior distribution of latent variable z, which is parameterized by a diagonal Gaussian distribution:

where mean θ(x) and variance diag(σ2 (x)) are both nonlinear functions of x parameterized with deep neural networks.

Reparameterization reparameterizes z as a function of θ and σ instead of using the standard sampling method. VAE introduces a standard Gaussian noise variable ✏ for generating samples from qφ(z ♣ x), namely the reprarameterization trick:

VAE uses an objective which encourages the model to keep its posterior distribution of z close to its prior distribution. And this objective is a valid lower bound estimation on the true log likelihood of the data, making VAE a generative model. The objective takes the following form:

Maximizing the objective function is equivalent to maximizing the reconstruction likelihood of observable variable x and minimizing the Kullback-Leibler divergence between the approximated posterior and the prior distribution of latent variable z.

3 Bilingual Variational Autoencoder

In this section, we introduce our proposed model in detail. Formally, given the definition in Eq.1, the variational lower bound of BiVAE can be formulated as follows:

where x and y denote observable semantic equivalent sentences from each language respectively, p(z) is the prior distribution of latent variable z, which is Gaussian distribution here. qφ(z ♣ x↪ y) is our posterior approximator, p✓(x ♣ z) and p✓(y ♣ z) represent the conditional distribution of x and y conditioned on z. ✓ and φ are parameters of generative and inference neural networks respectively. Our goal is to train high quality bilingual word features with the proposed model.

3.1 Variational Inference Network

Neural sentence encoder

The neural encoders(see Figure2) aim at encoding the parallel sentence pairs into distributed representations. Following [Bowman et al., 2015]'s approach, we adopt RNN to encode the parallel sentence pairs. But different from Bowman, who encodes the monolingual sentences using vanilla RNN, we adopt bidirectional LSTM as sentences are better summarized with context both forwards and backwards. Given an instance of parallel sequence pairs [wA 1 ↪wA 2 ↪ ▷▷▷↪wA TA ] and [wB 1 ↪wB 2 ↪ ▷▷▷↪wB TB ], the forward LSTM reads the sequence from left to right and the backward LSTM in the opposite direction:

where WA 2 R♣VA♣⇥dA and WB 2 R♣VB♣⇥dB are lookup matrix for words in language A and B respectively. ♣ VA ♣ and ♣ VB ♣ denote vocabulary size, dA and dB denote dimension of the word embeddings, TA and TB denote length of sequence A and B respectively. -!h iA, - h iA, -!h iB and - h iB are hidden representations at position i generated in two directions. We call WA and WB lookup matrix as they learn word features in separate vector spaces, we use them only for sentence summarization. Finally, we concatenate the hidden states at the last time step of each LSTM encoder to represent the sentences:

Variational neural inferer

Exactly modeling the posterior p(z ♣ x↪ y) is intractable. Conventional models typically employ the mean field approaches. However, due to its oversimplified assumption, it fails to capture the true posterior distribution of z. Following [Kingma and Welling, 2013], we employ neural networks to approximate the posterior distribution of z to get a tighter lower bound and assume the approximator has the following form:

where mean θ and standard variance σ are outputs of neural networks based on sentence representations hA and hB. Function f projects the separate representations of parallel sentence pairs onto our concerned latent semantic space:

where Wz 2 Rdz⇥(dhA +dhB ) and bz 2 Rdz are connection matrix and bias respectively, dz is the dimensionality of the latent space, dhA and dhB are dimensionality of outputs of variational encoders. g(·) is elementwise activation function which we set Relu throughout our experiment.

We obtain the diagonal Gaussian distribution parameter θ and σ2 through linear regression:

where θ, logσ2 are both dz dimension vectors.

To obtain representations of z, we employ the same technique as VAE and reparameterize it as:

During encoding, we reconstruct the parallel sentence pairs based on sampled z0 from qφ(z ♣ x↪ y)

3.2 Decoder Network

Given the latent variable z, decoder network defines the probability over the observable variables x and y. By sampling z0 from the posterior approximator qφ(z ♣ x↪ y), we are able to reconstruct x and y and estimate the expectation likelihood term. We represent the reconstructed parallel sentence pairs using one-hot BoW representations for the following reasons:

• ÷ By projecting the common distributed representations of parallel sentence pairs, namely z, to each language's vocabulary, where we introduce two connection matrix EA and EB which can be viewed as word embeddings, we obtain cross lingual representations of words embedded in exactly the same vector space.
• ÷ Acceleration. Large scale learning requires efficient approaches. As our goal is training cross lingual word embeddings instead of machine translation, we employ a softmax decoder by independently generating the words(z ! [¶xi♦↪ ¶yi♦]).
• ÷ Vanilla LSTM decoder is sensitive to subtle variation in the hidden states, thus making it hard to train the neural sentence encoder.

Inspired by [Miao et al., 2016], the conditional probability over observable variables x and y is modelled by multinomial logistic regression with parameter shared across sentence pairs:

where EA 2 Rdz⇥♣VA♣ and EB 2 Rdz⇥♣VB♣ learns jointspace semantic word embeddings and bA, bB represent the bias terms.

We use Monte Carlo method to estimate the expectation term over the posterior in Eq.6 which typically has less variance than the generic estimator. The training objective for an instance of parallel sentence pair (x↪ y) is defined as follows:

where z0 l = θ + σ ⊙ ✏l, ✏l ⇠ N(0↪ I). L is the number of samples.

The first term is the Kullback-Leibler divergence between the posterior and prior distribution of z, which can be integrated analytically[Kingma and Welling, 2013]. The KLdivergence term can be interpreted as regularizing φ, encouraging the approximate posterior to be close to the prior p(z). And the second term can be interpreted as the negative expected reconstruction error of x and y. Since the objective in Eq.14 is differentiable, we can jointly optimize the model parameters ✓ and φ using stochastic gradient optimization.

The model is implemented using Tensorflow[Abadi et al., 2016], the model parameters of both generative process and posterior estimator are trained jointly using ADAM[Kingma and Ba, 2014]. We also apply dropout and batch normalization[Srivastava et al., 2014; Ioffe and Szegedy, 2015] to the neural networks introduced in our model to reduce overfitting. We use 200 units for LSTM memory cell and 40 units for latent variable z, consequently 40 units for the word embeddings.

4 Experiment

In this section we present experiments which evaluate the quality of induced cross-lingual representations. We evaluate the embeddings in a standard cross-lingual document classification task which tests semantic transfer of information across languages.

4.1 Cross Lingual Document Classification

We use an exact replication of the cross-lingual document classification(CLDC) setup introduced by [Klementiev et al., 2012] to evaluate the embeddings. The CLDC task setup is as follows: A labeled data set of documents in some language A is available to train a classifier, however we are interested in classifying documents in another language B at test time. As Klementiev noted, the aim of this task is not to provide a state-of-the-art document classifier but rather to examine the validity of joint semantic space model. Specifically, we train an averaged perceptron classifier[Freund and Schapire, 1999] on the labelled training data in the source language and then attempt to apply the classifier to the target data. Documents are represented as the tf-idf weighted sum of the embedding vectors of the words that appear in the documents.

4.2 Dataset and Setup

As our joint space model utilizes parallel corpus only, we train the bilingual embeddings for the English-German language pair using Europarl v7 parallel corpus[Koehn, 2005], and use the induced representations to classify a subset of the English and German sections of the Reuters RCV1/RCV2 multilingual corpora[Lewis et al., 2004] that are assigned to only one of four categories: CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social), and MCAT (Markets).

We preprocess the corpus by lowercasing all the words, removing all the punctuations and replacing all the digits with 0. We select the top words by term frequency appearing in the corpus and keep the vocabulary size as ♣ V EN ♣=40000 and ♣ V DE ♣= 50000.

For the classification experiment, 15000 documents(for each language) were selected randomly by Klementiev[Klementiev et al., 2012] from RCV1/RCV2 corpus. One third of the selected documents(5000) were used as test sets and a varying size between 100 and 10000 of the remainder were used as training set. Another 1000 documents were kept as development set for hyper-parameter tuning. A multi class document classifier was trained for 10 epoch with English documents and is used to classify German documents and vice versa.

4.3 Results

Table1 summarizes results on the task of CLDC. We compare the performance of our model with some baselines and previous work. The Majority Class is a system where we simply classify the test documents as the class with the most training examples. The MT is a phrase-based machine translation system where test documents were translated into the same language as training documents. We also summarize some of the previous work. [Klementiev et al., 2012] proposed to train two neural network languages models simultaneously along with a regularization term that encourages pairs of frequently aligned words to have similar word embeddings. BiCVM[Hermann and Blunsom, 2013] learned word embeddings via minimizing the compositional representations between parallel sentence pairs. BAE[AP et al., 2014] reconstructed the bag-of-words representation of semantic equivalent sentence pairs to learn word embeddings. Bi-Skip[Luong et al., 2015] extended skip-gram model to bilingual circumstances where separate context of aligned word pairs were jointly predicted. [Shi et al., 2015] proposed a matrix cofactorization framework, CLSim, for learning cross lingual embeddings.

From Table1 we conclude that our proposed model outperforms most previous work and performs comparably with the previous state-of-the-art model, CLSim, with BiVAE tops the accuracy of submission de ! en. But note that our model is trained with Europarl corpus only while CLSim induce the embeddings using both Europarl and RCV corpus.

Different from those approaches which start from monolingual objective learning embeddings in separate vector space following cross lingual objective as constraints, BiVAE learns word representations embedded in exactly the same vector space without the help of explicit alignments. While on the other hand, one drawback of BiVAE is that the learned features can not be enhanced with monolingual corpus due to the structure of its probabilistic language model.

4.4 Qualitative Examples and Visualization

We find, for each English word, a list of top several English and German words closest to it based on Euclidean distance in a learned joint bilingual vector space. Our list of words includes ¶ january, oil, man, economy, microsoft, president, market, great, communication, law♦. Table 2 illustrates the properties captured within and across languages. Bilingually, our embeddings succeed in selecting the 1-best translations for all words in the list. Monolingually, our embeddings possess a clearly good clustering structure, which reveals topic nature of the semantic vector sapce.

Figure 3 gives a visualization of the joint vector space using t-SNE[Maaten and Hinton, 2008]. The English and German words which are translations of each other are represented by almost the same point in the vector space, revealing the semantic validity of the joint vector space.

5 RelatedWork

5.1 Cross-lingual Learning

Overall, approaches for training bilingual word embeddings can be categorized into several classes: offline mapping, monolingual adaption and parallel training.

In offline mapping, word representations are first trained on each language independently and a mapping is then learned to transform representations from one language to another. The advantage of this approach is its speed as no further training of word representations is required given monolingual word embeddings. Word embeddings trained in this approach include [Mikolov et al., 2013] which utilizes translation pairs

-20 -10

Figure 3: Visualization of joint semantic space using t-SNE. We represent 10 high frequency En-De word translation pairs.

to learn linear mapping.

Monolingual adaption jointly optimize the monolingual objectives of each language, with cross-lingual objective enforced as cross-lingual regularizer. One advantage of this approach is that it utilizes monolingual corpus in addition to parallel corpus to enhance the learned features.

Unlike previous schemes fix representations on either one or both sides, Parallel training leverage sentence aligned parallel corpus only and train a model to learn similar representations for aligned sentences. [Hermann and Blunsom, 2013; AP et al., 2014] and the model proposed in this paper follow this approach. One advantage of this approach is that models can be designed to train word representations embedded in exactly the same continuous vector space, avoiding explicit alignment.

As introduced in section 1, cross-lingual word embeddings can be applied to various NLP tasks, including semantic applications such as cross-lingual dictionary induction[Vulic and Moens, 2013a; Mikolov et al., 2013] and CLDC[Klementiev et al., 2012] as well as syntactic applications such as cross-lingual syntactic dependency parsing[T¨ackstr¨om et al., 2012] and lexicon extraction[Vulic and Moens, 2013b]. We prove our embeddings effective in CLDC task but its application remains to be explored in other cross-lingual NLP tasks.

5.2 Variational Neural Inference

In order to perform efficient inference and learning in generative probabilistic models on large-scale dataset, variational autoencoder was recently proposed by [Kingma and Welling, 2013; Rezende et al., 2014]. Different from conventional mean field approximation, VAE employs neural network to approximate the posterior distribution of latent variable and optimize the model parameters with a reparameterized variational lower bound using the stochastic gradient optimization technique. Following Kingma, semi-supervised VAE has been proposed to model labeled dataset. Variational RNN has been proposed to deal with sequential data, which have been proved successful in speech modeling.

Variational neural inference has also shown strong performance in text processing. [Miao et al., 2016] proposes a generic variational inference framework for generative and conditional models of text. [Bowman et al., 2015] imposes a prior distribution on the hidden states of a standard RNN language model, helping generating sentences from the latent semantic space. [Zhang et al., 2016] introduces a latent variable z to a standard neural machine translation framework to guide the generation of target translations. To the best of our knowledge, we are the first to introduce this technique to learn cross-lingual word embeddings.

6 Conclusion

In this paper, we propose a variational encoder-decoder framework for cross-lingual learning. By introducing the hidden variable z, we learn cross-lingual word embeddings in exactly the same continuous vector space instead of projecting them from separate spaces. We also conduct a standard CLDC task to evaluate BiVAE. Experiment results show that BiVAE performs comparably with the previous reported state-of-the-art model.

For future work, we are interested in modifying the variational neural decoder, e.g. LSTM decoder, to generate plausible parallel sentence pairs from the latent semantic space. With limited training corpus, we can train BiVAE to help generate more semantic equivalent sentence pairs to enrich the corpus in an unsupervised way.

Acknowledgements

This work is partially supported by the National High Technology Research and Development Program of China (Grant No. 2015AA015403). We would also like to thank the anonymous reviewers for their helpful comments.