Joint and Coupled Bilingual Topic Model Based Sentence Representations for Language Model Adaptation

1 Introduction

In recent years, LM adaptation for SMT tries to improve a generic LM by using smaller amounts of training data [Ruiz and Federico, 2011]. Selecting training data which are similar to the translation task from the large corpus has become an important approach to improve the performance of LM [Eck et al., 2004; Zhao et al., 2004; Masskey and Sethy, 2010; Axelrod et al., 2011; Lu et al., 2012b]. The bias LM, which is estimated with the similar training data, would empirically provide more accurate lexical probabilities, thus better match with the translation task [Lu et al., 2012b] and promote more fluent translations.

To select similar training data for LM adaptation in SMT, many researchers proposed various approaches, such as TFIDF [Eck et al., 2004; Zhao et al., 2004; Foster and Kuhn, 2007], centroid similarity [Masskey and Sethy, 2010], crossentropy difference [Axelrod et al., 2011], phrase-based similarity [Lu et al., 2012a], cross-lingual similarity (CLS) [Ananthakrishnan et al., 2011], and cross-lingual information retrieval [Snover et al., 2008; Lu et al., 2012b]. They perform at the word level, exact only term matching schemes. Unfortunately, they all do not consider the topic information and the distribution information of words in the whole LM training corpus. These information have been successfully used for LM adaptation in SMT [Tam et al., 2007; Zhao and Xing, 2007; Ruiz and Federico, 2011] and proved very useful. These approaches apply the topic posterior distribution to the target language LM via marginal adaptation or minimum discrimination information (MDI) adaptation. However, they focus on modifying the LM parameter itself, which is different from data selection based LM adaptation in the category.

In this paper, we argue that it is beneficial to capture wordtopic information and word-distribution information for data selection based LM adaptation. To this end, we present two more principled bilingual topic models (BLTM), joint BLTM (JBLTM) and coupled BLTM (CBLTM), and map the data selection task into cross-lingual semantic representations that are language independent. We apply these BLTM based sentence representations to cross-lingual data selection for LM adaptation in SMT. Unlike the general topic models, these two BLTMs work at the sentence level, represent a sentence as a distribution of semantic topics, and assume that a word in the sentence is generated from a mixture of these topics. They learn a semantic representation from bilingual parallel corpus which shares the same or similar topic fractions as much as possible. Compared with the traditional approaches, our approaches are potentially more effective because they consider the topic information and the distribution of words into similar data selection. It is thus reasonable to expect that using such information as ranking features is likely to further improve the quality of selected sentences and adapted LMs, as we will show in the experiments. To the best of our knowledge, this is the first extensive and empirical study of applying BLTM based sentence representations into cross-lingual data selection for LM Adaptation.

Specifically, we make following contributions:

• We present two BLTMs, JBLTM and CBLTM, and formulate the data selection task into cross-lingual semantic representations at the sentence level that are language independent (in section 4.1 and 4.2). JBLTM assigns a pair

of corresponding sentences having the same topic distributions, while CBLTM assigns the similar topic distributions.

• We apply JBLTM and CBLTM based sentence semantic representations to cross-lingual data selection (in section 4.3). Then, we rank and select the candidate sentences in the target LM training corpus for a sentence in the translation task by the semantics-based likelihood.
• To further improve the quality of selected sentences, we introduce a linear ranking model framework in which other information (e.g., word-based translation model [Lu et al., 2012b]) is incorporated as features (in section 4.4).
• Finally, we conduct large-scale experiments on LM perplexity and translation performance, respectively (in section 5). The results demonstrate that our approach significantly outperforms the state-of-the-art approaches.

The remainder of this paper is organized as follows. Section 2 introduces the related work. Section 3 describes the framework of cross-lingual data selection for LM adaptation. Section 4 presents our BLTM based sentence representations for cross-lingual data selection. Section 5 gives large-scale experiments, and followed by the conclusions in section 6.

2 RelatedWork

Most previous work focus on monolingual LM adaptation in SMT based on multi-pass translation. They select the sentences which are similar to the translation hypotheses. However, the noisy translation hypotheses would mislead data selection process, and degrades the performance of adapted LM. To address this problem, Lu et al. (2012b) propose cross-lingual data selection based LM adaptation for SMT, which models data selection based on the translation task directly. Following this convincing idea, we present BLTM based cross-lingual data selection for LM adaptation.

Recently, topic models have been extended from monolingual to handle cross-lingual or multi-lingual cases, where there are pairs or tuples of corresponding documents in different languages.

Tam et al. (2007) propose a bilingual-LSA approach for LM adaptation. This model consists of two hierarchical LDA models, constructed from parallel corpora, and assumes that the topic distributions of the source and target documents are identical. A one-to-one correspondence between LDA models is enforced by learning the hyperparameters of the variational Dirichlet posteriors in one LDA model and bootstrapping the second model by fixing the hyperparameters. HM-BiTAM [Zhao and Xing, 2007] constructs a generative model in which words from a target language are sampled from a mixture of topics drawn from a Dirichlet distribution. It generates unigram LMs for both the source and target language and thus can be used for LM adaptation through MDI. Ruiz and Federico (2011) presents a bilingual latent semantic model for LM adaptation by combining text in the source and target language into very short documents and performing PLSA during model training. During inference, documents containing only the source language can be used to infer a full topic-word distribution on all words in the target language's vocabulary, from which they performMDI adaptation on a background LM. However, the above approaches focus on modifying the LM parameter itself, which is different from data selection based LM adaptation in the category.

Except the application for LM adaptation, some crosslingual topic models are proposed for other NLP tasks. Polylingual topic model (PLTM) [Mimno et al., 2009] is an extension to LDA that views documents in a tuple as having a shared topic distribution. Joint PLSA (JPLSA) is a variant of PLTM when documents of different languages share the same word-topic distribution, and coupled PLSA (CPLSA) extends JPLSA by constraining corresponding documents to have similar fractions of words assigned to each topic according to the posterior distribution of topic assignments, instead of sharing the prior topic distributions [Platt et al., 2010]. Clickthrough-based bilingual latent semantic model is proposed for web search [Gao et al., 2011], and it can be viewed as a special case of PLTM, where search queries and web documents are assumed to be written in two different languages.

Our presented two BLTMs are the variants and extensions to these previous models. For the first time, we work the topic model at the sentence level, and strive to effectively learn model parameters from bilingual parallel corpus for the application of LM adaptation in SMT.

3 Cross-Lingual Data Selection Based Language Model Adaptation

Our LM adaptation is an unsupervised similar data selection guided by BLTM based bilingual sentence representations. For the source sentences in the translation task, we estimate a new bias LM, from the corresponding target LM training sentences which are selected as the similar sentences. Following [Zhao et al., 2004; Snover et al., 2008], the generic LM Pg(wi|h) and the bias LM Pb(wi|h) are combined as the adapted LM Pa(wi|h), as follows,

where the interpolation factor µ is estimated using the Powell Search algorithm [Press et al., 1992].

Our work focuses on the two BLTM based bilingual sentence representations and their application for cross-lingual data selection in SMT.

4 BLTMBased Sentence Representations for Cross-Lingual Data Selection

In this section, we will introduce two BLTMs (JBLTM and CBLTM) in detail. Furthermore, we will introduce JBLTM and CBLTM based sentence representations for cross-lingual data selection for LM adaptation in SMT.

4.1 JBLTMBased Sentence Representations

The first JBLTMmodel (Figure 1(a)), which is a close variant of JPLSA [Platt et al., 2010], assumes that a pair of parallel sentences have a common topic distribution θ, and uses this assumption to optimize the probability of the data.

We assume that a sentence U = u1, · · · , u|U| in the translation task and its paired similar sentence V = v1, · · · , v|V | in the LM training corpus share a common topic distribution, but use different vocabularies to express these topics. Formally, inspired by [Gao et al., 2011], JBLTM assumes the following process of generating a cross-lingual sentence pair. For each topic z, a pair of different word distributions (φUz , φVz ) are selected from a Dirichlet prior with concentration parameter β, where φUz is a topic-specific word distribution of U, and φVz a topic-specific word distribution of V . Assuming there are T topics, we have two sets of distributions φU = (φU1 , · · · , φUT ) and φV = (φV1 , · · · , φVT ) . Then, for each U and its paired V , a topic distribution θ is drawn from a Dirichlet prior with concentration parameter α. Each word in U is then generated by first selecting a topic z according to θ, and drawing a word from φUz . The paired sentence V has the corresponding derivation.

Thus, the semantics-based log-likelihood of a similar cross-lingual sentence pair, together with the paired wordtopic vectors, is

where

Recently, MAP inference has shown to perform comparably to the best inference method for LDA , if the hyperparameters are chosen optimally for the inference method [Asuncion et al., 2009]. Therefore, we use the standard EM algorithm [Dempster et al., 1977] to estimate the parameter (θ, φU , φV ) of JBLTM by maximizing the joint loglikelihood of the parallel corpus and the parameters for LM adaptation.

E-Step: computing the posterior probability for each word u in U and each word v in its paired sentence V for the latent variables z, as follows,

M-Step: updating the parameters for given posterior probability computed in the previous E-step, as follows,

where, α, βU and βV are hyper-parameters, each corresponding to one Dirichlet prior. DU and DV are the size of the source and target vocabulary, respectively. Let n(u,U) be the frequency of u in U, n(v, V ) be the frequency of v in V , and we have the following definitions,

4.2 CBLTMBased Sentence Representations

For the task of similar sentence selection based LM adaptation, we also want our BLTM to assign similar topic distributions θ to a pair of corresponding sentences, which is different from the first JBLTMmodel. This difference becomes especially apparent when corresponding sentences have different lengths in our task for LM adaptation. In this case, the model will tend to derive a topic vector θ which explains the longer sentence best, making the sum of the two sentences' log-likelihoods higher. Modeling the shorter sentence's best topic carries little weight.

To address the above problems, we present the second BLTM, CBLTM (Figure 1(b)). CBLTM models both sentences equally, and the topic vectors of a pair of sentences in two languages are shown completely independent, which is somewhat similar to CPLSA [Platt et al., 2010]. Compared with [Gao et al., 2011], this modification not only solves the above problems, but also significantly improves the selected sentences for LM adaptation.

CBTLM is also trained using a modified EM algorithm, and we use posterior regularization [Graca et al., 2008; Ganchev et al., 2010] to place linear constraints on the expectations of closer topic assignments to two corresponding sentences.

E-Step: computing the posterior distributions of topics from parallel corpus like Equation (4) and (5), but with some corresponding changes, as follows,

Then, the posterior distributions are projected onto a constrained set of distributions, for which the expected fraction of tokens in U that are assigned topic t is similar with the expected fraction of tokens in V .

M-Step: updating the expected counts with respect to this projected posterior distribution like Equation (6) to (8), but with some corresponding changes for Equation (8) to (10), as follows,

Next, we describe how the projection is performed. For a sentence pair (U, V ), we would like CBLTM to be such that the expected fraction of tokens in U that get assigned topic t is approximately the same as the expected fraction of tokens in V that get assigned the same topic t, for each topic t = 1 · · · T . This is exactly what we need to make each pair of corresponding sentences close. Let (U, V ) be a pair of sequences of tokens and their topic assignments, where

Let Q denotes the posterior distribution set over the hidden topic assignments,

Q′ is an ideal distribution set that has the desired property,

such that, the expected fraction of each topic is similar or closer in U and V ,

Then, the projection minimizes the KL divergence between two sets of distributions Q and Q′ , KL(Q′ ||Q). The projection can be formulated as a constrained optimization problem, where we seek an ideal set of distributions Q′ that is closest to Q,

The valid ideal distribution space Q is non-empty, and the problem of Equation (22) can be solved efficiently in its dual form. The final corpus-wide objective is summed over sentence-pairs, and also contains terms for the probabilities of the parameters θ and φ given the Dirichlet priors. The norm of ϵ is minimized, which makes the expected proportions of topics in two sentences as close as possible.

We initialize the models deterministically by assigning each word to exactly one topic to begin with, such that all topics have roughly the same number of words. Words are sorted by frequency and thus words of similar frequency are more likely to be assigned to the same topic.

4.3 Ranking Candidate Sentences

In our two BLTMs, the topics are generated from V written in the target language, and the word u is generated from topicspecific word distributions in the source language. Therefore, it can be considered as performing a translation of two corresponding sentences from the translation task to the large LM training corpus via hidden topics. Then, we can rank the candidate sentences in the target LM training corpus for a sentence in the translation task by the semantics-based likelihood, and further select the top-N ranked similar sentences to adapt LM.

In our experiments for LM adaptation, we use the following sentence ranking function,

where, γ is the tuning parameter. P(u|CU) is the unsmoothed background model. CU refers to the translation task, |CU| refers to its size, and n(u,CU) refers to the frequency of u in CU , respectively.

4.4 Linear RankingModel

To further improve the performance of ranking the candidate sentences, we introduce a linear ranking model framework for cross-lingual data selection in which different models are incorporated as features.

We consider the linear ranking model as follows,

where the model has a set of N features. Each feature is an arbitrary function that maps (U, V ) to a real value, and λn is the corresponding parameter. We optimize the parameter using the Powell Search algorithm [Press et al., 1992] via cross-validation.

In our experiments, we use word translation model based cross-lingual data selection (CLWTM) [Lu et al., 2012b] as the features, listed as follows,

where, P(u|v) is the word-to-word based translation model. P(v|V ) is the un-smoothed sentence model, and n(v, V ) refer to the frequency of v in V .

5 Experiments and Results

We measure the utility of our approach in two ways: (a) comparing reference translations based perplexity of adapted LMs with the generic LM, and (b) comparing translation performance of adapted LMs with the generic LM.

5.1 Experimental Setup

We conduct experiments on the following NIST ChineseEnglish translation tasks. The bilingual corpus comes from LDC 1 , which consists of 3.4M sentence pairs with 64M/70M Chinese/English words. The LM training corpus is the English Gigaword corpus 2 , which consists of 11.3M sentences. Our first test set (test-1) is NIST 2006 MT Evaluation test set, our second test set (test-2) is NIST 2008 MT Evaluation test set, and our tuning set is NIST 2005 MT Evaluation test set.

To improve the efficiency of BLTM training, we consider the data sets in our task are constructed by the key words or important words. We adopt a variant of TextRank algorithm [Mihalcea and Tarau, 2004] for key word extraction which achieves state-of-the-art accuracy, and manually remain 75% of total words as the important words.

For the two BLTMs training, we both use 100 topics. We use α = 1.1 and β = 1.01 in JBLTM, and α = 1.1 and βU = βV = 1.01 in CBLTM for the values of the concentration parameters, respectively. We perform learning through MAP inference using EM (with a projection step for CBLTM). We do up to 300 iterations, and do early stopping based on task performance on the tuning set. The JBLTM model requires more iterations before reaching its peak accuracy, tending to require around 200 to 250 iterations for convergence. CBLTM requires fewer iterations, but each iteration is slower due to the projection step.

We randomly divide the tuning set into five subsets and conduct 5-fold cross-validation experiments. In each trial, we tune the parameter µ in Equation (1) and parameter λ in Equation (27) with four of five subsets, respectively. Then, we apply these parameters to one remaining subset.

For comparable experiments, we choose TF-IDF as a foundation since it has gain the state-of-the-art performance for monolingual LM adaptation [Eck et al., 2004; Zhao et al., 2004; Foster and Kuhn, 2007]. CLSs [Lu et al., 2012b] is the improved algorithm on CLS [Ananthakrishnan et al., 2011] with optimization measure like TF-IDF. CLWTM refers to word-to-word translation model based cross-lingual data selection [Lu et al., 2012b], and it is the state-of-theart cross-lingual data selection approach for LM adaptation. JBLTM+CLWTM and CBLTM+CLWTM are our presented approaches which are incorporated with CLWTM by the linear ranking model.

5.2 Perplexity Analysis

We estimate the generic 4-gram LM with the entire LM training corpus as the baseline. Then, we select the top-N ranked sentences which are similar to the tuning set, estimate the bias 4-gram LMs with these selected sentences, and interpolate with the generic 4-gram LM as the adapted LMs. All the LMs are estimated by the SRILM toolkit [Stolcke, 2002] with interpolated modified Kneser-Ney discounting.

Figure 2 shows the English reference translations based perplexity of adapted LMs vs. the size of selected data. Obviously, proper size of similar sentences with the translation task makes the adapted LM perform well, but if too many noisy data are taken into the selected sentences, the performance significantly becomes worse. Furthermore, our approaches have two clear advantages. JBLTM and CBLTM are more stable than other approaches, and the increasing trends of perplexity after the certain size are relatively slow. JBLTM+CLWTM and CBLTM+CLWTM have more obvious perplexity reductions than other approaches.

According to the perplexity results in Figure 2, we select the top 16K sentences for test-1 and test-2 which are similar to the test set for adapting LM, respectively. From Table 1, we can see that our approaches have significant perplexity reduction on the test set compared with other approaches, and the results indicate that adapted LMs are significantly better predictors of the corresponding translation task at hand than the generic LM. Next, we use these adapted LMs for translation experiments to show the detailed performance of selected training data for LM adaptation.

5.3 Translation Experiments

We conduct translation experiments by hierarchical phrasebased (HPB) [Chiang, 2007] translation system, and evaluated the translation quality by BLEU-4 score [Papineni et al., 2002]. The generic LM and adapted LMs are estimated as above in perplexity analysis experiments. We use minimum error rate training [Och, 2003] to tune the feature weights of HPB on the tuning set.

Table 2 shows the main translation results, and the improvements are statistically significant at the 95% confidence interval with respect to the baseline. From the results, we get some clear trends:

(1) CLSs outperforms TF-IDF (row 3 vs. row2), so data selection based LM adaptation in SMT should be performed by the cross-lingual model.

(2) CBLTM outperforms JBLTM (row 6 vs. row 5). Compared to JBLTM, CBLTM assigns similar topic distributions θ to a pair of corresponding sentences and models both corresponding sentences equally when they have different lengths. Thus, CBLTM is more suitable for cross-lingual data selection based LM adaptation, and it obtains better performance.

(3) CBLTM outperforms CLWTM (row 6 vs. row 4). CLWTM is the state-of-the-art cross-lingual data selection method for LM adaptation, it learns the word-to-word translation probability from parallel corpus. However, it ignores the word-topic information and the word-distribution information for similar data selection, and these information are proved useful for LM adaptation.

(4) JBLTM+CLWTM significantly outperforms JBLTM and CLWTM (row 7 vs. row 5 and 4), CBLTM+CLWTM significantly outperforms CBLTM and CLWTM (row 8 vs. row 6 and 4), respectively. This demonstrates that the wordtopic information learned from BLTM (JBLTM and CBLTM) and the word-to-word translation probability learned from CLWTM are complementary to each other for cross-lingual data selection, and the performance of adapted LMs are further improved by incorporating them together with the linear ranking model.

5.4 The Effectiveness of Data Selection

Previous work for LM adaptation can be divided into two categories: one focuses on selecting similar training data (e.g., all approaches in the above experiments) and the other focuses on modifying LM parameter itself (e.g., bilingual-LSA [Tam et al., 2007]). In this section, we choose bilingual-LSA as the foundation since it has gained the state-of-the-art performance, and compare these two categories' effectiveness. Table 3 shows the effectiveness of data selection to the performance of LM adaptation. We can see that CBLTM and JBLTM significantly outperform bilingual-LSA for LM adaptation. Data selection based LM adaptation introduces similar training data, and this is more directly for the translation task at hand. We suspect this leads to the poor performance of bilingual-LSA. However, it will be instructive to explore the detail of these two categories' effectiveness in future.

6 Conclusions

In this paper, we present two more principled BLTMs, JBLTM and CBLTM, and apply the BLTM based sentence representations into cross-lingual data selection for LM adaptation in SMT. Unlike the general topic models, our two BLTMs work at the sentence level, and represent a sentence as a distribution of semantic topics. Compared to the traditional approaches, our approaches are potentially more effective because they consider the word-topic and worddistribution information into the similar data selection. Furthermore, we introduce a linear ranking framework which can incorporate different models for cross-lingual data selection to further improve the performance of adapted LMs. Largescale experiments are conducted on LM perplexity and translation performance, respectively, and the results demonstrate that our approaches significantly outperform the state-of-theart approaches for LM adaptation.

Acknowledgments

This work was supported by 863 program in China (No. 2011AA01A207). We thank Dr. Zhenbiao Chen and Dr. Hongyan Li for his helpful discussions and suggestions, and also thank the anonymous reviewers for their insightful and helpful comments.