A Subspace Learning Framework for Cross-Lingual Sentiment Classification with Partial Parallel Data

1 Introduction

With the development of web 2.0, more and more user generated sentiment data have been shared on the web. They exist in the form of user reviews on shopping or opinion sites, in posts of blogs or customer feedback in different languages. These labeled user generated sentiment data are considered as the most valuable resources for the sentiment classification task. However, such resources in different languages are very imbalanced. Manually labeling each individual language is a time-consuming and labor-intensive job, which makes crosslingual sentiment classification essential for this application.

Cross-lingual sentiment classification aims to automatically predict sentiment polarity (e.g., positive or negative) of data in a label-scarce target language by exploiting labeled data from a label-rich language. The fundamental challenge of cross-lingual learning stems from a lack of overlap between the feature spaces of the source language data and that of the target language data. To address this challenge, previous work in the literature mainly relies on automatic machine translation engines [James G. Shanahan, 2004; Wu et al., 2008; Wan, 2009; Prettenhofer and Stein, 2010; Wan, 2009; Pan et al., 2011; Xiao and Guo, 2013]. Most of these studies translate documents from the source language to the target language or vice versa, and then apply the standard monolingual classification methods. However, due to the difference in language and culture, there exists a word drift problem. That is, while a word frequently appears in one language, its translated version may rarely appear in the other language [Guo, 2012].

Another group of works propose to use a large amount of bilingual parallel corpora to induce language-independent representations [Lu et al., 2011; Meng et al., 2012; Klementiev et al., 2012]. In many real applications, however, it is often the case that we have some partial parallel data but it is a expensive and time-consuming job to acquire large amount of document-aligned (or sentence-aligned) data on different languages. For example in multilingual sentiment classification data [Prettenhofer and Stein, 2010], we have a large amount of Amazon product reviews written in four languages (English, French, German and Japanese) but only 2000 unlabeled parallel reviews between English and each of the other three languages are existed.

To handle such data, we propose a novel subspace learning approach to address the cross-lingual sentiment classification with the partial parallel data. Our assumption is that document-aligned parallel data describe the same semantics in two different languages, they should share the same latent representations under the discriminative subspace regarding the same classification task. Then the language gap between the source language and the target language can be reduced via the shared representations. Specially, our model is achieved by jointly learning the document-aligned review data and un-aligned data from the source language and the target language via a non-negative matrix factorization framework [Lee and Seung, 1999]. We derive an efficient algorithm for learning the factorization and provide proof of convergence. To evaluate the effectiveness of the proposed approach, we conduct a set of experiments with cross-lingual sentiment classification tasks on multilingual Amazon product reviews. The empirical results show that the proposed approach is effective for cross-lingual sentiment classification and outperforms other comparison methods.

The remainder of this paper is organized as follows. Section 2 introduces the related work. Section 3 describes our proposed subspace learning framework on partial parallel data. Section 4 presents the experimental results. Finally, we conclude this paper in Section 5.

2 RelatedWork

In this Section, we present the related work on traditional monolingual sentiment classification and cross-lingual sentiment classification.

2.1 Sentiment Classification

Sentiment classification has gained widely interest in natural language processing (NLP) community. Methods for automatically classifying sentiments expressed in products and movie reviews can roughly be divided into supervised and unsupervised (or semi-supervised) sentiment analysis. Supervised techniques have been proved promising and widely used in sentiment classification [Pang et al., 2002; Pang and Lee, 2008; Liu, 2012]. However, the performance of these methods relies on manually labeled training data. In some cases, the labeling work may be time-consuming and expensive. This motivates the problem of learning robust sentiment classification via an unsupervised (or semi-supervised) paradigm.

The most representative way to perform semi-supervised paradigm is to employ partial labeled data to guide the sentiment classification [Goldberg and Zhu, 2006; Sindhwani and Melville, 2008; Li et al., 2011]. However, we do not have any labeled data at hand in many situations, which makes the unsupervised paradigm possible. The most representative way to perform unsupervised paradigm is to use a sentiment lexicon to guide the sentiment classification [Turney, 2002; Taboada et al., 2011; Zhou et al., 2014b] or learn sentiment orientation of a word from its semantically related words mined from the lexicon [Peng and Park, 2011]. Sentiment polarity of a word is obtained from off-the-shelf sentiment lexicon, the overall sentiment polarity of a document is computed as the summation of sentiment scores of the words in the document. All these works focus on the monolingual sentiment classification, we point the readers to recent books [Pang and Lee, 2008; Liu, 2012] for an in-depth survey of literature on sentiment classification.

2.2 Cross-Lingual Sentiment Classification

Cross-lingual sentiment classification aims to automatically predict sentiment polarity (e.g., positive or negative) of data in a label-scarce target language by exploiting labeled data from a label-rich language. The fundamental challenge of cross-lingual learning stems from a lack of overlap between the feature spaces of the source language data and that of the target language data.

To bridge the language gap, previous work in the literature mainly relies on machine translation engines or bilingual lexicons to directly adapt labeled data from the source language to the target language. Banea et al. [2008] employed the machine translation engines to bridge the language gap in different languages for multilingual subjectivity analysis. Wan [2009] proposed a co-training method to train Chinese sentiment classification model on English labeled data and their Chinese translations. English labeled data are first translated into Chinese, and then the bi-view sentiment classifiers are trained on English and Chinese labeled data respectively. Pan et al. [2011] proposed a bi-view non-negative matrix tri-factorization (BNMTF) model for cross-lingual sentiment classification problem. They employed machine translation engines so that both training and test data are able to have two representations, one in a source language and the other in a target language. The proposed model is derived from the non-negative matrix factorization models in both languages in order to make more accurate prediction. Prettenhofer and Stein [2010] proposed a cross-lingual structural correspondence learning (CL-SCL) method to induce languageindependent features. Instead of using machine translation engines to translate labeled text, the authors first selected a subsect of pivot features in the source language to translate them into the target language, and then use these pivot pairs to induce cross-lingual representations by modeling the correlations between pivot features and non-pivot features in an unsupervised fashion. However, due to the difference in language and culture, these translation-based methods cause a word drift problem. That is, while a word frequently appears in one language, its translated version may rarely appear in the other language [Guo, 2012].

Another group of works propose to use a large amount of unlabeled parallel corpora to induce language-independent representations [Lu et al., 2011; Meng et al., 2012; Klementiev et al., 2012]. Meng et al. [2012] proposed a generative cross-lingual mixture model (CLMM) to learn previously unseen sentiment words from the large bilingual parallel data. Klementiev et al. [2012] proposed to induce the bilingual word embeddings by using neural language models in an unsupervised setting. Zhou et al. [2014a] proposed to learn the distributed semantics for sentiment classification. A common property of these approaches are that they need a large amount of unlabeled parallel corpora to train the word alignment model [Meng et al., 2012; Klementiev et al., 2012]. In many real applications, however, it is often the case that we have some partial parallel data but it is an expensive and time-consuming job to acquire large amount of parallel data on different languages. In this paper, we propose a novel subspace learning framework for cross-lingual sentiment classification. Our proposed approach only needs some partial parallel data and can jointly learn the document-aligned review data and un-aligned data from the source language and the target language.

3 Our Proposed Approach

In this section, we present the formulation and optimization of our proposed subspace learning framework on the partial parallel data for cross-lingual sentiment classification. In the cross-lingual setting, a partial parallel data set X = {X(s,t) ,X(s) ,X(t)} is given, where X(s,t) = [(xs1 , xt1 ); · · · ; (xsc , xtc )] denotes the documents present and only present in the parallel data, X(s) = [x(s) c+1 ; · · · ; x(s) c+m] denotes the documents from the source language, and X(t) = [x(t) c+m+1 ; · · · ; x(t) c+m+n ] denotes the documents from the target language, c, m and n represent the number of documents present and only present in the parallel data, from the source language and the target language. The goal of cross-lingual sentiment classification with partial parallel data is to establish a latent subspace where the documents from the parallel data should share the same latent representations. The language gap can be reduced with the help of the shared latent representations.

3.1 Model Formulation

Let X(s,t) = [ ˆX(s) c , ˆX(t) c ] be composed of documents ˆX(s) c ∈ Rc×d1 , ˆX(t) c ∈ Rc×d2 coming from the the parallel sentiment review data. We now have the documents of each language denoted as ¯X(s) = [ ˆX(s) c ;X(s) ] ∈ R(c+m)×d1 , ¯X(t) = [ ˆX(t) c ;X(t) ] ∈ R(c+n)×d2 . As we deal with the text data, non-negative matrix factorization (NMF) [Lee and Seung, 1999] has been widely used for document representation, which actually assumes that the documents are generated by additive combination of an underlying set of hidden units. Incorporating the NMF technique into our problem, for each language, its latent subspace learning can be formulated as:

where U(s) ∈ Rk×d1 and U(t) ∈ Rk×d2 are the basis matrix for each language's latent space, and ¯V(s) = [ ˆV(s) c ;V(s) ] ∈ R(c+m)×k , ¯V(t) = [ ˆV(t) c ;V(t) ] ∈ R(c+n)×k are the latent representation of documents in the latent space. The same latent dimension k is shared between the source language and the target language. λ is the tradeoff parameter for the regularization term Ω( ¯V). By equation (1) and equation (2), the latent space basis U and the corresponding document latent representation V are simultaneously learned to minimize the document reconstruction error, which forces all documents from each language to share the similar compact representation in the latent space.

So far, the latent spaces are learned independently for each language. For the cross-lingual setting on the partial parallel data, the document-aligned sentiment review data in ˆX(s) c , ˆX(t) c describing the same data object, their latent representation ˆV(s) c , ˆV(t) c should also be the same. Combining this idea and equation (1), equation (2), by enforcing ˆV(s) c = ˆV(t) c = ˆVc, we aim to minimize the following objective function:

Now we can have the homogeneous feature representation for all documents as V = [ ˆVc;V(s) ;V(t) ], whether they are originally partial or not. Any standard monolingual sentiment classification approach can be applied to such representation. The novelty of equation (3) lies in that ¯V(s) and ¯V(t) share the same part ˆVc and at the same time has their own individual part V(s) and V(t) . Moreover, the proposed framework is learned by using all available documents of each language, the individual basis matrix U(s) and U(t) are connected by the common ˆVc. Lasso is used for Ω(V) in this work as one of the mostly often used regularization for text analysis [Hu et al., 2013].

3.2 Learning Algorithm

In this subsection, we present the solution to the optimization problem in equation (3), which is convex in latent representations V given the basis matrix U and vice versa, but not jointly convex in both, we propose an iterative update procedure and prove its convergence. Firstly, the basis matrices are initialized by the initialization step and then the following steps are repeated until convergence: (1) minimizing the objective function O over V with fixed U; and (2) minimizing the objective function O over U with fixed V.

Initialization Since the efficiency of the iterative optimization procedure is greatly affected by the initialization step, we learn the initial value of U rather than random selection:

From equation (4), we can see that U(s) and U(t) are essentially initialized by applying traditional NMF on documents without partial parallel data. This initialization is also solved by iterative optimization. At each iteration, Oinit is minimized alternatively over ˆVc ≥ 0, U(s) and U(t) . Fixing ˆVc ≥ 0, U(s) and U(t) can be independently optimized by:

Fixing U(s) and U(t) , ˆVc is optimized by:

(1) Minimizing O over V with fixed U Given the basis matrix U for each language, the computation of ˆV do not depend on each other. Therefore, the objective function in equation (3) reduces to:

Noting the same formulation of equation (10) as equation (7), so at the first iteration, ˆVc has already been obtained from initialization.

(2) Minimizing O over U with fixed V Given V = [ ˆVc;V(s) ;V(t) ] , the latent representations for documents of each language can be obtain as [ ˆVc;V(s) ] and [ ˆVc;V(t) ], minimizing O over U now independently reduces to:

To solve the optimization problem in equation (5)∼ equation (12), which are lasso regularized NMF with one factor fixed, we employ the greedy coordinate descent (GCD) approach proposed by Hsieh and Dhillon [2011], which is about 10 times faster than cyclic coordinate descent scheme and proved to converge. To get a robust and stable sparsity tradeoff parameter for different data sets, λ is normalized by the data size during the optimization, i.e., for equations (8), (9) and (10), λ is timed by coefficient (d1 + d2)/k, d1/k and d2/k, respectively.

Theorem 1. The objective function in equation (3) is nonincreasing under the optimization procedure in Algorithm 1. Lemma 1. [Hsieh and Dhillon, 2011] For least squares NMF, if a basis matrix, latent representation pair sequence {(Uj,Vj)} is generated by GCD, then every limit point of this sequence is a stationary point.

Proof. To prove Theorem 1, we only need to prove that the objective function in equation (3) is non-increasing after each step. With fixed U(s) and U(t) , the objective function value of equation (3) with respect to ˆVc,V(s) ,V(t) equals the sum of the objective function value of equation (8)∼equation (10). With fixed ˆVc,V(s) ,V(t) , the objective value of equation (3) with respect to U(s) , U(t) equals the sum of equation (11)∼equation (12). By Lemma 1, the objective function value of equation (8)∼equation (12) are guaranteed to converge to some local minima. So the objective function value of equation (3) is guaranteed to non-increase after each step.

3.3 Cross-Lingual Sentiment Classification

Once the parameters { ˆVc, V(s) , V(t)} and {U(s) , U(t) } are obtained, we use the learned latent representations V(s) and V(t) of the labeled data from the two languages and train a simple sentiment classification model using a linear support vector machine (e.g., Liblinear [Fan et al., 2008]). Then, we predict the sentiment polarity of the test data from the target language on the latent representations V(t) with the learned classification model.

4 Experiments

In this section, we conduct experiments for cross-lingual sentiment classification on multilingual Amazon product reviews in four languages.

4.1 Data Sets

We use the multilingual sentiment classification data set provided by [Prettenhofer and Stein, 2010], which contains Amazon product reviews in four languages (English (E), French (F), German (G) and Japanese (J)) of three categories (Books (B), DVD (D), Music (M)). The English product reviews are sampled from previous cross-domain sentiment classification data sets [Blitzer et al., 2007], while the other three language product reviews are crawled from Amazon by the authors in November. For each category of the product reviews, there are 2000 positive and 2000 negative English reviews (represented with X(s) ), and 1000 positive and 1000 negative reviews for each of the other three languages (represented with X(t) ). Besides, we have another 2000 unlabeled parallel sentiment reviews between English and each of the other three languages (partial parallel data, represented with X(s,t) ). All data described in the above is publicly available from the website. 1

Following the literature [Xiao and Guo, 2013], each review is represented as a unigram bag-of-words vector representation and each entry is computed with tf-idf. Given the four languages from the three categories, we construct 18 crosslingual sentiment classification tasks (EFB, EFD, EFM, EGB, EGD, EGM, EJB, EJD, EJM, FEB, FED, FEM, GEB, GED, GEM, JEB, JED, JEM) between English and the other three languages. For example, the task EFB uses English Books reivews as the source language data and uses French Books reviews as the target language data.

4.2 Baseline Algorithms

In our experiments, we re-implement the following baseline algorithms in order to compare with the proposed approach:

• TB: This is a target bag-of-words baseline method, which trains a supervised monolingual classifier on the labeled training data from the target language without using the unlabeled parallel data.
• CL-LSA: This is the cross-lingual learning method described in [Gliozzo and Strapparava, 2006], which first translates each document from one language into the other language via a bilingual dictionary to produce augmenting features, and then performs latent semantic analysis (LSA) over the augmented bilingual documentterm matrix.
• CL-SCL: This is the cross-lingual structural correspondence learning (SCL) method described in [Prettenhofer and Stein, 2010], which first chooses some pivot features and then automatically induce the cross-lingual correspondences with a bilingual dictionary. We implement this approach by using the source codes provided by the authors. 2
• CL-MT: This is a machine translation based method, which first uses the Google Translate (http://translate.google.com) tool to translate the target language documents into the source language and

1 http://www.webis.de/research/corpora/

2 https://github.com/pprett/nut

then trains a monolingual classifier with labeled training data from both languages.

• CL-OPCA: This is the cross-lingual oriented principal component analysis (OPCA) method described in [Platt et al., 2010], which first learn cross-lingual representations with all data from both languages by performing OPCA and then train a monolingual classifier with labeled data from both languages in the induced feature space.
• CL-TS: This is the state-of-the-art method for crosslingual sentiment classification described in [Xiao and Guo, 2013], which formulates the cross-lingual representation as a matrix completion problem to infer unobserved feature values of the concatenated documentterm matrix in the space of unified vocabulary set from the source and target languages by using unlabeled parallel bilingual documents. 3

In all experiments, we train the sentiment classification model on the latent representations using the Liblinear [Fan et al., 2008]. For CL-LSA, CL-OPCA and CL-TS, we use the same parameter value k = 50 as suggested in the paper [Xiao and Guo, 2013]. For CL-SCL, we use the same parameter setting as suggested in the paper [Prettenhofer and Stein, 2010]: the number of pivot features is set as 450, the threshold value for selecting pivot features is 30, and the reduced dimensionality after singular value decomposition is 100. We choose the above parameter values empirically because these parameter settings have shown superior performance on the same benchmark [Prettenhofer and Stein, 2010].

3 There is a slight exception: using the same data set and parameter setting, our re-implemented systems have slight differences with the results reported in [Xiao and Guo, 2013].

4.3 Classification Accuracy

For each of the 18 cross-lingual sentiment classification task, we use all documents from the two languages (including 2000 parallel sentiment data). Then we use all documents from the source language and randomly choose 100 documents from the target language as labeled data to build the classification model, and use the rest data from the target language as test data. For our proposed algorithm, we choose the latent dimension k values from {10, 20, 50, 100, 150, 250, 300}, choose the parameter λ value from {10-5 , 10-4 , 10-3 , 10-2 , 10-1 , 1}. We perform the model parameter selection by running the algorithm 3 times based on random selection of 100 labeled target training data (in short, Tuning set). Finally, we set parameters k = 100 and λ = 10-2 . We then use the selected model parameters for all the 18 tasks and run each experiment for 10 times based on random selections of 100 labeled target documents (in short, Test set). 4 The average sentiment classification accuracies and standard deviations are presented in Table 1.

From Table 1, we can see that our proposed approach outperforms all other six comparison methods in general. The target baseline TB performs poorly on all the 18 tasks, which indicates that 100 labeled documents from the target language is far from enough to obtain an accurate and robust sentiment classifier in the target language domain. All the other five cross-lingual sentiment classification methods, CL-LSA, CL-SCL, CL-MT, CL-OPCA and CL-TS, consistently outperform the baseline method TB across all the 18 tasks, which demonstrates that the labeled training data from the source language domain is useful for classifying the target language data. Nevertheless, the improvements achieved by these five methods over the baseline are much smaller than the proposed approach. Among all the 18 tasks, our proposed approach increases the average test accuracy over the baseline TB by at least 9.21% on the EJM task and up to 15.30% on the EFB task. Moreover, our propose approach also outperforms CLLSA, CL-SCL, CL-MT and CL-OPCA across all the 18 tasks, outperforms CL-TS on 17 out of the 18 tasks and achieves the slight lower performance than CL-TS on the GEB task.

4 During the experiments, we try our best to eliminate the overlap between the Tuning set and Test set. In this paper, we would do several experiments by random sampling rather than perform cross-validation in order to make a fair comparison with the previous work [Prettenhofer and Stein, 2010; Xiao and Guo, 2013].

We also conduct significance tests for our proposed approach and the state-of-the-art CL-TS using a McNemar paired test for labeling disagreements [Gillick and Cox, 1989]. The results in bold formate indicate that they are significant with p < 0.05. All these results demonstrate the efficacy and robustness of the proposed subspace learning framework on partial parallel data for cross-lingual sentiment classification.

4.4 Convergence Analysis

In section 3.2, we have shown that the proposed objective function is convergent. Here, we empirically show the convergence analysis for the selected four tasks due to space limitation. Figure 1 shows the convergence curves of our proposed algorithm on the four tasks. From the figures, y-axis is the value of the objective function and x-axis denotes the iteration number. It can be seen that the objective function value monotonically decreases as the iteration number increases. Though it takes a lot of rounds to converge, the GCD in each iteration runs very fast, usually within 60 iterations.

4.5 Parameter Study

In this section, we explore the effect on the parameters k and λ to the sentiment classification performance. We choose k value from {10, 20, 50, 100, 150, 250, 300} and λ value from {10-5 , 10-4 , 10-3 , 10-2 , 10-1 , 1}. Due to space limitation, we only present the experimental study on the selected six tasks (EFB, EFD, EFM, EJB, EJD and EJM) in Figure 2. From Figure 2, we can see that the proposed algorithm achieves stable good performance when k is around 100 and λ is around 10-2 .

Figure 2: Influence of k and λ on the selected six tasks.

5 Conclusion

In this paper, we propose a novel subspace learning framework by leveraging the partial parallel data for cross-lingual sentiment classification. The proposed approach is achieved by jointly learning the document-aligned review data and unaligned data from the source language and the target language via a non-negative matrix factorization framework. We also derive an efficient algorithm for learning the factorization and provide proof of convergence. To evaluate the effectiveness of the proposed approach, we conduct a set of experiments with cross-lingual sentiment classification tasks on multilingual Amazon product reviews. The empirical results show that the proposed approach is effective for cross-lingual sentiment classification and outperforms other comparison methods. In the future, we will study how to extend the proposed subspace learning framework to multilingual settings and to nonlinear latent subspace cases.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 61303180 and No. 61272332), the Beijing Natural Science Foundation (No. 4144087), the Major Project of National Social Science Found (No. 12&2D223), the Fundamental Research Funds for the Central Universities (No. CCNU15ZD003), and also Sponsored by CCF-Tencent Open Research Fund. We thank the anonymous reviewers for their insightful comments.