Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model

1 Introduction

Recent progress in deep learning benefits a variety of NLP tasks and leads to significant performance improvements when large-scale annotated datasets are available. For highresource languages, e.g., English, it is feasible for many tasks to collect sufficient labeled data to build deep neural models. However, for many languages, there might not exist enough data in most cases to make full use of the advances of deep neural models. As such, various cross-lingual transfer learning methods have been proposed to utilize labeled data from high-resource languages to construct deep models for low-resource languages [Kim et al., 2019; Lin et al., 2019; He et al., 2019; Vuli´c et al., 2019]. Nonetheless, most crosslingual transfer learning research focuses on mitigating the discrimination of languages, while leaving the domain gap less explored. In this study, we concentrate on a more challenging setting, i.e., cross-lingual and cross-domain (CLCD) transfer, where in-domain labeled data in the source language is not available.

∗This work was supported by Alibaba Group through Alibaba Innovative Research (AIR) Program.

†This work was done when Juntao Li was an intern at the National University of Singapore.

Conventionally, cross-lingual methods mainly rely on extracting language-invariant features from data to transfer knowledge learned from the source language to the target language. One straightforward method is weight sharing, which directly reuses the model parameters trained on the source language to the target language, by mapping an input text to a shared embedding space beforehand. However, previous research [Chen et al., 2018] revealed that weight sharing is not sufficient for extracting language-invariant features that can generalize well across languages. As a result, a languageadversarial training strategy was proposed to extract invariant features across languages, using non-parallel unlabeled texts from each language. Such a strategy performs well for the bilingual transfer setting but is not suitable for extracting language-invariant features from multiple languages, since features shared by all source languages might be too sparse to retain useful information.

Recently, pretrained cross-lingual language models at scale, e.g., multilingual BERT [Devlin et al., 2019] and XLM [Conneau and Lample, 2019; Conneau et al., 2019], show very competitive performance over various cross-lingual tasks, and even outperform pretrained monolingual models on low-resource languages. Through employing parallel texts (unlabeled for any specific task) and shared subword vocabulary over all languages, these pretrained crosslingual models can effectively encode input texts from multiple languages to one single representation space, which is a feature space shared by multiple languages (more than one hundred). While generalizing well for extracting languageinvariant features, cross-lingual pretraining methods have no specific strategy for extracting domain-invariant features. In our CLCD setting, both language-invariant and domaininvariant features need to be extracted.

To address the aforementioned limitation of cross-lingual pretrained models [Conneau et al., 2019] in CLCD scenarios, we propose an unsupervised feature decomposition (UFD) method, which only leverages unlabeled data in the source language. Specifically, our proposed method is inspired by the recently proposed unsupervised representation learning method [Hjelm et al., 2019] and can simultaneously extract domain-invariant features and domain-specific features by combining mutual information maximization and minimization. Compared to previous cross-lingual transfer learning methods, our proposed model maintains the merits of crosslingual pretrained models, i.e., generalizing well for over a hundred languages, and only needs unlabeled data in the source language for domain adaptation, which is suitable for more cross-lingual transfer scenarios.

We evaluate our model on a benchmark cross-lingual sentiment classification dataset, i.e., Amazon Review [Prettenhofer and Stein, 2010], which involves multiple languages and domains. Experimental results indicate that, with the enhancement of the pretrained XLM cross-lingual language model, our proposed UFD model (trained on some unlabeled raw texts in the source language) along with a simple linear classifier (trained on a small labeled dataset in the source language and the source domain) outperforms state-of-the-art models that have access to strong cross-lingual supervision (e.g., commercial MT systems) or labeled datasets in multiple source languages. Furthermore, incorporating our proposed UFD strategy with an unlabeled set of 150K instances in the source language leads to continuous gains over the strong pretrained XLMmodel that is trained on one hundred languages and terabytes of texts. Extensive experiments further demonstrate that unsupervised feature decomposition on a pretrained cross-lingual language model outperforms a pretrained domain-specific language model trained on over 100 million sentences.

2 RelatedWork

Cross-lingual transfer learning (CLTL) has long been investigated [Yarowsky et al., 2001] and is still one of the frontiers of natural language processing [Chen et al., 2019]. Through utilizing rich annotated data in high-resource languages, CLTL significantly alleviates the challenge of scarce training data in low-resource languages. Conventionally, CLTL mainly focuses on resources that are available for transferring, e.g., collecting parallel texts between two languages to directly transfer model built in a rich-resource language to a low-resource one [Pham et al., 2015] or constructing annotated data in the target language by machine translation systems [Xu and Yang, 2017]. Subsequently, with the success of deep learning, cross-lingual word embeddings are proposed to learn the shared representation space at the fundamental level and can benefit various downstream tasks [Artetxe et al., 2018; Conneau et al., 2018a]. Later, a crosslingual sentence representation is also proposed [Conneau et al., 2018b]. Chen et al. [2018] designed a languageadversarial training strategy to extract language-invariant features that can directly transfer to the target language.

Another direction is pretraining cross-lingual [Conneau and Lample, 2019] or multilingual language models [Devlin et al., 2019]. Benefiting from the large-scale training texts and model size, these pretraining methods have changed the face of cross-lingual transfer learning. Empirical results demonstrate that representation space shared by one hundred languages can significantly outperform the language-specific pretrained models [Conneau et al., 2019]. As language adversarial training will lead to sparse language-invariant representations when multiple languages are involved [Chen et al., 2019], we follow the line of cross-lingual language model pretraining. Unlike previous pretraining methods, we focus on domain adaptation of these pretrained models. To maintain the generalization ability of the cross-lingual pretrained model, we mainly consider the unsupervised domain adaptation setting. The work most related to ours is proposed for unsupervised representation learning [Hjelm et al., 2019], which is primarily used for visual representation learning.

3 Model

In this section, we first define the problem discussed in this paper and then describe the proposed method in detail.

3.1 ProblemDefinition &Model Overview

In this paper, we consider a setting where we only have a labeled set Ds,s of a specific language and a specific domain which we call source language and source domain, and we want to train a classifier to be tested on a set Dt,t of a different language and a different domain which we call target language and target domain. We also assume access to some unlabeled raw data Ds,u of multiple domains including the target domain from the source language during the training phase, which is usually feasible in practical applications. We call this setting unsupervised cross-lingual and cross-domain (CLCD) adaptation.

As illustrated in Figure 1, the proposed method consists of three components: a pretrained multilingual embedding module which embeds the input document into a languageinvariant representation, an unsupervised feature decomposition (UFD) module which extracts domain-invariant features and domain-specific features from the entangled languageinvariant representation, and a task-specific module trained on the extracted domain-invariant and domain-specific features. We adopt XLM 1 [Conneau and Lample, 2019] as the multilingual embedding module in our method, which has been pretrained by large-scale parallel and monolingual data from various languages and is the current state-of-theart cross-lingual language model. We describe the other two modules and the training process in the following subsections.

3.2 Unsupervised Feature Decomposition

Mutual Information Estimation

Before elaborating on the proposed unsupervised feature decomposition module, we first present some preliminary knowledge on mutual information estimation, which is employed in the training objectives of UFD. Mutual information (MI) is growing in popularity as an objective function in unsupervised representation learning. It measures how informative one variable is of another variable. In the context of unsupervised representation learning, MI maximization is usually adopted such that the encoded representation maximally encodes information of the original data. MI is difficult to compute, particularly in continuous and high-dimensional settings, and therefore various estimation approaches have been proposed.

1 The latest version XLM-R is adopted, which is trained on over one hundred languages and 2.5 terabytes of texts.

In our method, we adopt a recently proposed neural estimation approach [Belghazi et al., 2018], which estimates MI of two continuous random variables X and Y by training a network to distinguish between samples coming from their joint distribution, J, and the product of their marginal distributions, M. This estimation utilizes a lower-bound of MI based on the Donsker-Varadhan representation (DV) of KL-divergence [Donsker and Varadhan, 1983],

where Tω is a discrimination function parameterized by a neural network with learnable parameters ω. It maps a sample from space X × Y to a real value in R. Through maximizing ̂ I DV , Tω is encouraged to distinguish between samples drawn from J and M by assigning the former large values while the latter small ones.

Proposed Method

Let X ∈ Rd denote the language-invariant representation generated by the pretrained multilingual embedding module. It is then fed into the proposed UFD module as input. As shown in Figure 1, we introduce two feature extractors: the domain-invariant extractor Fs (i.e., the two-layer feedforward network with ReLU activation on the left), and the domainspecific extractor Fp (i.e., the two-layer network on the right).

We denote the extracted features as Fs(X) and Fp(X) respectively. Note that for Fs, we add residual connections to better maintain domain-invariant attributes from X.

Specifically, Fs aims to extract domain-invariant features from the language-invariant representation in an unsupervised manner. Since the multilingual embedding module is pretrained on open domain datasets from over one hundred languages, presumably the generated language-invariant representations should contain certain attributes that can be generalized across domains. When Fs is trained on multiple domains with jointly maximizing MI between the inputs and outputs of each domain, it is encouraged to retain the shared features among those domains from the language-invariant representations. In this way, Fs is forced to pass domaininvariant information fromX to Fs(X).

We utilize the neural network-based estimator as presented in Equation (1) for computing MI. In our case, as Fs(X) is dependent on X, we can simplify the DV-based MI estimator to a Jensen-Shannon MI estimator as suggested in [Hjelm et al., 2019]:

where x is an input embedding with empirical probability distribution P. As Fs(x) is directly computed from x, (x, Fs(x)) can be regarded as a sample drawn from the joint distribution of X and Fs(X). x′ corresponds to an input embedding from ˜P = P, i.e., x′ is computed from a random sample drawn from the same input distribution, such that (x′ ,Fs(x)) is drawn from the product of marginal distributions. sp(z) = log(1+ez ) is the softplus activation function. The training objective of Fs is to maximize the MI on X and Fs(X) and the loss is formulated as follows:

where ωs denotes the parameters of the discrimination network in the estimator and ψs denotes the parameters of Fs. To facilitate learning of domain-invariant features, we also propose to maximize the MI on Fs(X) and the corresponding intermediate representation (first layer output) F ′s (X), and the training loss is as follows:

where ωr denotes the parameters of the discriminator network in the estimator.

Recall that the objective of Fp is to extract domain-specific features, which is supposed to be exclusive and independent of domain-invariant features. We propose to minimize the MI between features extracted by Fs and Fp, and the training loss is formulated as follows:

where ψp denotes the parameters of Fp. ωp denotes the parameters of the discrimination network in MI estimator.

The training objective of the proposed UFD component is thus to minimize the overall loss as follows:

where α, β, and γ are hyperparameters to balance the effects of sub-losses.

3.3 Task-Specific Module

In the task-specific module, we first employ a linear layer that maps the concatenation of the domain-invariant and domainspecific features in R2d into a vector representation in Rd . A simple feedforward layer with softmax activation is then employed on this mapped vector representation to output the task label. We train this module on Ds,s and the cross-entropy loss denoted as Lt is utilized as the training objective.

3.4 Training

Note that the parameters of the multilingual embedding module are pretrained and set to be frozen in the entire training process. We first optimize the parameters of UFD, i.e., {̂ωs, ̂ωr, ̂ωp, ̂ ψs, ̂ ψp} by minimizing LUFD on the unlabeled set Ds,u. Once the UFD module is trained, we fix its parameters and train the task-specific module by minimizing Lt on the labeled set Ds,s.

4 Experimental Setting

4.1 Datasets

We conduct experiments on the multi-lingual and multidomain Amazon review dataset [Prettenhofer and Stein, 2010], which serves as a benchmark in previous cross-lingual sentiment analysis research and also supports cross-lingual and cross-domain evaluation. This dataset includes texts in four languages, i.e., English, German, French, and Japanese, and each language contains three domains, i.e., Books, DVD, and Music. There are a training set and a test set for each domain in each language and both consist of 1,000 positive reviews and 1,000 negative reviews.

In our CLCD evaluation, we treat English as the only source language and attempt to adapt to the other three languages. As each language contains three domains, we can construct 3 × 2 CLCD source-target pairs between English and a specific target language. Therefore, we have 18 CLCD source-target pairs in total considering all three target languages. During training, we first utilize some unlabeled raw data from the source language for optimizing the proposed UFD. Then, the training set from the source language and source domain is used for training the task-specific module. During testing, the model is evaluated on the test set of the target language and target domain.

We draw samples from 3 larger unannotated datasets of Books, DVD, and Music domains released in [He and McAuley, 2016]. The statistics of the three datasets are given in Table 1. We randomly sample 50K documents from each domain as the unlabeled domain-specific set in the source language (i.e., English) to be utilized during training. To encourage Fs to capture domain-shared features, we utilize domainspecific unlabeled sets from all domains (50K × 3) in training the UFD module. We also show the change in performance when varying the number of unlabeled samples in Section 5.

4.2 Baselines

We denote our proposed model as XLM-UFD, and we compare it with the following baselines:

CL-RL [Xiao and Guo, 2013] is a cross-lingual word representation learning method, which learns the connection between two languages by sharing part of the word vectors.

Bi-PV [Pham et al., 2015] attempts to learn paragraph vectors in a bilingual context setting by sharing the distributed representations of unannotated parallel data from different languages.

CLDFA [Xu and Yang, 2017] is a cross-lingual distillation method which leverages a parallel corpus of documents. An adversarial feature adaptation strategy is applied for reducing the mismatch between the labeled data and the unlabeled parallel document.

MAN-MOE [Chen et al., 2019] addresses the multilingual transfer setting, i.e., there are multiple source languages with labeled data. Building upon a languageadversarial training module, this model utilizes a mixture-ofexperts (MOE) module to dynamically combine private features of different languages.

The above four baselines were originally proposed for adaptation in a cross-lingual setting, e.g., adapting from English-Books to German-Books. We report their official results released in the original papers, which can be regarded as upper bounds for their CLCD performances. Note that the setting of MAN-MOE is different, where N to 1 adaption is performed, i.e., from N source languages to one target language. Thus, its cross-lingual performance cannot be simply viewed as the upper bound of its CLCD performance. We retrain the model in the CLCD setting as another baseline described later. For the baselines described below, they are all trained in the CLCD setting.

ADAN [Chen et al., 2018] exploits adversarial training to reduce the representation discrepancy between the encoded source and target embeddings.

MAN-MOE-D is the version of MAN-MOE trained in a CLCD setting. As this specific model performs N to 1 adaptation, it can adapt from multiple source domains from the same source language to a specific target domain and target language. In our experiments, MAN-MOE-D utilizes two source domains from the same source language. For example, when the target language and domain are German-Books, MAN-MOE-D takes labeled set from both English-DVD and English-Music during training.

Multi-BPE combines the pretrained multilingual bytepair embeddings in 275 languages [Heinzerling and Strube, 2018] 2 with the task-specific classifier used in our proposed model to perform CLCD adaptation. This model is used to calibrate the performance of the subword embeddings shared across multiple languages.

2 https://nlp.h-its.org/bpemb/multi/

DLM is a pretrained domain-specific language model 3 implemented with the code of XLM 4 [Conneau and Lample, 2019]. It employs the pretrained multilingual byte-pair embeddings as the initialized representations of input texts to mitigate the gap between the source language and the target language. This model is used to study the effect of leveraging large scale domain-specific unlabeled texts.

XLM refers to the model where we simply add a feedforward layer with softmax activation as the output layer on top of pretrained XLM [Conneau et al., 2019].

4.3 Training Details

The hidden dimension of XLM is 1024. The input and output dimensions of the feedforward layers in both Fs and Fp are 1024. The discriminator of Tωs , Tωr , and Tωp share the same model structure as suggested in previous work [Hjelm et al., 2019], i.e., the discriminator consists of two feedforward layers with ReLU activation. The input and output dimensions of the first feedforward layer in the discriminator are 2048 and 1024. The input and output dimensions of the second feedforward layer are 1024 and 1. The input dimension of the single-layer task-specific classifiers is 1024. All trainable parameters are initialized from a uniform distribution [-0.1, 0.1].

We utilize 100 labeled data in the target language and target domain as the validation set, which is used for hyperparameter tuning and model selection during training. The hyperparameters are tuned on the validation set of a specific sourcetarget pair, and are then fixed in all experiments of XLMUFD. Specifically, both UFD and the task-specific module are optimized by Adam [Kingma and Ba, 2014] with a learning rate of 1 × 10-4 . The batch size of training UFD and the task-specific module are set to 16 and 8, respectively. The weights α, β, γ in Equation (6) are set to 1, 0.3, and 1, respectively. During training, the model that achieves the best performance (lowest loss) on the validation set is saved for evaluation purpose.

3 Trained with the datasets presented in Table 1.

4 https://github.com/facebookresearch/XLM

5 Results

Table 2 presents the model comparison results and Table 3 shows the results of different ablation tests on XLM-UFD. Classification accuracy is used as the evaluation metric.

5.1 Model Comparison

In Table 2, the top 4 models are trained in a cross-lingual setting, and the middle 6 models are trained in a CLCD setting. We repeat the experiment on each source-target pair for 3 times with different random seeds and record the average result on each pair. Each reported result for models trained in the CLCD setting is the average result of the adaptation performance from two source domains in English. For example, a result under German-Books is the average of adaptation accuracies from English-DVD and English-Music.

We make the following observations from Table 2. (1) XLM-UFD achieves significantly better results over all baselines across all settings. It even substantially outperforms baselines trained in a cross-lingual setting with parallel texts from source and target languages such as CLDFA, which is a much less challenging setting. (2) One interesting finding is that MAN-MOE-D performs better than MAN-MOE. One possible reason is that MAN-MOE involves multiple source languages while invariant features shared by multiple languages might be too sparse to maintain enough information for extracting task-specific features. (3) Among the pretrained models, multilingual byte pair embeddings (MultiBPE) only achieves low performance. With the enhancement of large-scale domain-specific unlabeled text, the domainspecific language model (DLM) taking the multilingual byte pair embeddings as input obtains observable performance gains but still has much room for improvement. Benefiting from the large-scale training data and network size, XLM is able to perform better than the state-of-the-art task-specific models such as CLDFA and MAN-MOE-D on French and Japanese. When combined with the proposed UFD, significant performance gains are observed on XLM. This points out that domain adaptation is necessary for pretrained multilingual language models when applied to a specific task.

5.2 Ablation Study

To determine the effect of each module of XLM-UFD, we conduct a thorough model ablation. As presented in Table 3, we first examine the domain-invariant feature extractor along with MI maximization between the languageinvariant features from the multilingual embedding module and the extracted domain-invariant features, namely MaxMI. Classification accuracy shows that Max-MI with only domain-invariant features enhances the performance of XLM on German but leads to decreased performance on French and Japanese. Through supplementing the domain-specific feature extractor and the Min-MI objective (i.e., Lp), MaxMin-MI has a noticeable performance increase over MaxMI and outperforms XLM, which confirms that unsupervised feature decomposition can support dynamic domain-specific and domain-invariant feature combination and improve the task performance. With the enhancement of the intermediate Max-MI objective (i.e., Lr) between the intermediate features and the output of domain-invariant feature extractor, 2MaxMin-MI achieves significant performance improvement over Max-MI, and it is used as the full model for conducting other comparison and ablation. In addition, we investigate the effect of unlabeled data size of the source language used during training. It can be seen from Table 3 that the setting with 5K×3 unlabeled raw texts already yields a very promising performance. Further increasing the number of unlabeled examples continuously improves the model performance on French and German. When the unlabeled data size is larger than 10K×3, the performance improvement becomes marginal on Japanese but continues on German and French.

5.3 Visualization

To intuitively understand the process of domain-invariant feature and domain-specific feature extraction, we also give the t-SNE plots [Maaten and Hinton, 2008] of the UFD module at the tenth epoch. Specifically, we sample five thousand raw texts from the source domain and target language. Each raw text is processed by XLM, and the following domaininvariant feature extractor and domain-specific feature extractor, respectively. As presented in Figure 2, each data point in the plots represents an input text. We can observe from the left plot that the domain-invariant features and domainspecific features of input texts have a clear border that can be distinguished, which suggests that mutual information minimization can force the two extractors to exclusively extract two sets of features. The right plot in Figure 2 demonstrates that the domain-invariant features and the language-invariant representations from XLM are partly entangled, which can be explained by the fact that maximizing mutual information between them can force the domain-invariant extractor to retain useful features from the language-invariant representations that are shared among different domains.

6 Conclusions and FutureWork

In this paper, we propose a simple but effective unsupervised feature decomposition module that extends the pretrained cross-lingual model to a more useful CLCD scenario. Through introducing the mutual information maximization and minimization objectives in representation learning, our proposed method can automatically extract domaininvariant and domain-specific features from the languageinvariant cross-lingual space, by using only a small unlabeled dataset from the source language during training. Experimental results indicate that, with the enhancement of the proposed module, the cross-lingual language model XLM achieves continuous improvements, which leads to new stateof-the-art results on the Amazon review benchmark dataset in a CLCD setting. In the future, we will explore the effect of our proposed unsupervised feature decomposition model on other pretrained models and downstream tasks.

Acknowledgments

The authors gratefully acknowledge the assistance of Qingyu Tan in this work.