Distributional Correspondence Indexing for Cross-Lingual and Cross-Domain Sentiment Classification (Extended Abstract)

1 Introduction

Automated text classification methods usually rely on a training set of labelled examples in order to learn a classifier that will predict the classes of unlabelled documents. Deploying a model for a new target domain in the absence of high-quality annotated examples thus entails a substantial human labelling effort. Transfer learning (TL) [Pan and Yang, 2010] focuses on alleviating this problem by leveraging training examples from a different, although related, source domain for which the amount of available labelled examples is higher. TL thus operates in applicative scenarios in which the so-called "iid assumpution" (the training and the test data are randomly drawn from the same distribution) no longer holds.

One such scenario of the utmost importance is sentiment classification [Liu, 2012], the task of classifying opinionladen documents as conveying a positive or a negative sentiment towards a given entity (e.g., a product, a policy, a political candidate). In many contexts the amount of available, prelabelled opinions is scarce, or even null, in the case of new entities (e.g., products, policies, or political candidates). In such cases, promptly generating a sentiment classifier might become difficult, due to the considerable cost and time involved in producing a representative set of training documents.

∗This paper is an extended abstract of an article appeared as [Moreo Fern´andez et al., 2016].

In sentiment classification, TL finds a natural application in domain adaptation (DA), i.e., the task of adapting a sentiment classifier to operate on a new domain. For example, we might want to use a training set of book reviews written in English to classify movie reviews written in English, or to classify book reviews written in German. The former case is typically known as cross-domain adaptation, while the second one is instead known as cross-lingual adaptation [Pan et al., 2012].

Central to text classification is the representation of words and documents as numerical vectors that reflect, through their relative distances, the semantic relations among them, i.e., the smaller the distance between a pair of word vectors, the more semantically similar the words are assumed to be. While distributional semantic models like Latent Semantic Analysis (LSA) [Deerwester et al., 1990] and Latent Dirichlet Allocation (LDA) [Blei et al., 2003] have proven useful to represent similar words (e.g., beautiful and nice) close to each other (e.g., in terms of cosine similarity) in the vector space, one main difficulty in domain adaptation is to discover semantic correspondences that may exist across domains (e.g., writer and director when adapting from a source domain of books to a target domain of films).

Structural Correspondence Learning (SCL) was first adapted for the cross-domain case [Blitzer et al., 2007] and then extended to the cross-lingual case in [Prettenhofer and Stein, 2011]. SCL discovers correspondences among terms from different domains that show similar distributions with respect to a set of pivot terms (highly predictive terms expected to behave in a similar way in both domains). Each pivot defines in SCL an auxiliary classification problem that brings to bear "structural" information of the adaptation problem. The intuition according to which the semantics of words is somehow determined by its distribution in text with respect to other terms is generally referred to as the distributional hypothesis [Harris, 1954]. The Distributional Correspondence Indexing (DCI) [Moreo Fern´andez et al., 2016] method we propose also builds upon the distributional hypothesis, but it formalizes the notion of correspondence through the Distributional Correspondence Functions (DCF), which are much lighter, in terms of computational cost, than the auxiliary problems of SCL.

We experimentally show that DCI compares favourably to the state of the art in two popular sentiment datasets covering both cross-domain and cross-lingual adaptation, and at a substantially smaller computational cost.

2 Distributional Correspondence Functions

DCFs are a family of real-valued functions that quantify the degree of correspondence between two features (terms, in our case) f i and f j by comparing their context distribution vectors f i and f j from any (unlabelled) collection U. A context distribution vector is a unit-length n-dimensional vector that models how a feature relates to a set of contexts (documents, in our case); f i c denotes the value of the vector for term f i in context c, with f i c = 0 if f i does not appear in context c. The cases in which f i c > 0 are determined by the weighting function in use (e.g., tfidf), and might lead to different interpretations of the DCF, e.g., as a probability function in an event space (Section 2.1), or as a kernel in a vector space (Section 2.2).

A DCF is thus a function η : Rn × Rn → R, where the sign of η(f i , f j ) indicates the polarity of the correspondence, i.e., positive values indicate positive correlation and negative values indicate negative correlation. We force DCFs to be centered at 0 for null correspondence, i.e., η(f i , f j ) = 0 means the correspondence between f i and f j is not different than the correspondence between any pair of context vectors which randomly distribute the same number of events as f i and f j .

2.1 Probability-Based DCF

Probability-based DCFs derive from information theory, and build upon the joint probability distributions of two features in the binomial event space.

The first part of Table 1 shows the probability-based DCFs we investigate. We consider Pointwise Mutual Information (PMI - the ratio between the joint distribution and the product of the marginal distributions), and a simple probabilistic function (here called Linear) that contrasts the probabilities of f i conditioned on f j and f j , respectively. We also consider Mutual Information (MI - the reduction in entropy of a distribution due to the observation of another distribution) in an asymmetric version called AMI, where in order to distinguish between positive and negative correspondence we swap the sign of MI by means of a function ρ if tpr + tnr < 1.

2.2 Kernel-Based DCFs

Kernel-based DCFs are rooted in the kernel functions typically used, e.g., within Support Vector Machines (SVM). In this case the values in the context vector can be numeric, thus indicating the relative importance of a term in a given context (we use tfidf as the weighting function). We consider normalized context vectors, i.e., after weighting the documentby-term matrix we normalize the term vectors to unit length.

However (and differently from the probability-based DCFs we have considered above), not all kernels satisfy the null correspondence property. In general, a valid DCF ηK can be defined from a kernel K(·, ·) as:

where u′ and v′ are any two random context vectors with the same prevalences as u and v (random variables represented by U and V ), respectively.

The second part of Table 1 shows the Kernel-based DCFs we have considered, based on the cosine, the polynomial and the radial basis function kernels.

3 Distributional Correspondence Indexing

The working hypothesis of DCI is that words that play similar roles in their respective domains might present approximately invariant correspondences to the pivots. For example, consider the source domain of book reviews and the target domain of movie reviews. Consider also the list of pivots [intriguing, annoying, captivating, . . .] and a DCF η. We might thus expect to have:

etc., since book and movie terms play analogous roles in the source and target domains. Therefore, by embedding each term with respect to its DCF values to the pivots, similar terms across domains might end up being represented by similar vectors in a common vector space. The cross-lingual adaptation is achieved by presenting each pivot by a translation equivalent in the target language.

3.1 Pivot Selection

[Blitzer et al., 2006; 2007] defined pivots as highly predictive terms which occur frequently in the source and target domains and behave similarly in both domains. A good pivot should be highly task-dependent, and also present a similar degree of domain-dependence in the two domains. We thus look for pivots by selecting the top n terms (where n is a parameter) ranked by their pivot strength Ψ(f i ) defined as:

where MIs(f i ) is the mutual information (as previously done in [Prettenhofer and Stein, 2011]) of the term f i to the label (to be estimated on the training set Trs), and the rightmost factor is the cross-consistency, measured as the drift in prevalence (pi) of the term f i across the two domains.

3.2 TermProfiles

Given n pivots and a DCF η, each source and target term (including the pivot terms) is embedded as an n-dimensional profile vector

where f and pi are the context distribution vectors (typically estimated in unlabelled collections) of the term f being profiled and the ith pivot, respectively.

In order to avoid pivots with high prevalence to generate high DCF values which could lead to dominant dimensions, we center each profile dimension on its expected value and then rescale by the standard deviation, so that the values for all profile dimensions are approximately normally distributed in N(0, 1), and then rescale them to unit length.

As we assume pivot terms behave similarly in the two domains, we unify their term profiles by averaging the source and the target profiles in order to correct the possible misalignment between the source and target views of the pivot.

3.3 Document Indexing

Finally, train and test documents are indexed in the profile space via a weighted sum of all profile vectors associated to their terms. That is, document dj is represented as the mdimensional vector

where wij is the weight of term fi in document dj according to any weighting function (in our experiments we used the standard cosine-normalized tfidf), and ⃗ f ′ i is the normalized term profile vector for fi.

4 Experiments

In this section we experimentally compare DCI, equipped with different DCFs, to other state-of-the-art methods proposed in the literature.

We test these methods on two popular, publicly available sentiment datasets: Multi-Domain Sentiment Dataset 1 (MDS) [Blitzer et al., 2007] and Webis-CLS-10 2 [Prettenhofer and Stein, 2011]. MDS is frequently used for evaluating cross-domain adaptation approaches and consists of English product reviews taken from Amazon.com for the four domains Books, DVDs, Electronics, and Kitchen appliances. The dataset comprises 1000 positive reviews and 1000 negative reviews for each of the four domains, and a set of unlabelled documents ranging from 3,586 to 5,945 documents for each domain. According to the same evaluation procedure followed by the proposers of other methods we compare against, we randomly split each labelled dataset into a training set of 1600 instances and a test set of 400 instances. In order to facilitate reproducibility and to allow for a fair comparison with the results reported in previous literature, we use the same pre-processed version of the dataset and the same experimental protocol used in [Blitzer et al., 2007].

1 http://www.cs.jhu.edu/ mdredze/datasets/sentiment/

2 https://goo.gl/eppCro

The Webis-CLS-10 dataset, frequently used for evaluating cross-lingual methods, consists of Amazon product reviews written in four languages (English, German, French, and Japanese), covering three product domains (Books, DVDs, and Music). For each language-domain pair there are 2,000 training documents, 2,000 test documents, and from 9,000 to 50,000 unlabelled documents depending on the languagedomain combination. We adhere to the exact experimental protocol described in [Prettenhofer and Stein, 2011] and used by many researchers in the field.

As the evaluation measure we use standard accuracy (the proportion of correctly classified documents over the total number of outcomes), following the practice common in the related literature.

For this extended abstract, we have updated the list of baseline methods by adding new approaches (TCT [Huang et al., 2017], TrAdaB [Huang et al., 2017], DANN [Ganin et al., 2016], CL-TS [Zhou et al., 2015], Bi-PV [Xu and Wan, 2017], BiDRL [Zhou et al., 2016b], WSDNNs, [Zhou et al., 2016a], CLDFA [Xu and Yang, 2017]) which have been published in the cross-domain and cross-lingual arena after our original work [Moreo Fern´andez et al., 2016], and kept those which performed best in our original evaluation (SCLMI [Blitzer et al., 2007], SFA [Pan et al., 2010], SDA [Glorot et al., 2011], and SSMC [Xiao and Guo, 2014]). We also consider an upper bound that trains the classifier on the training set of the target domain ("Upper"), and a lower bound that trains the classifier on the source domain and then applies the trained classifier directly in the target domain, i.e., without carrying out any sort of knowledge transfer ("NoTrans"). For cross-lingual adaptation we also report the machine translation baseline ("MT"), which first translates all target documents into the source language (English, in our experiments) before giving them as input to the classifier; we use the pre-translated documents provided by [Prettenhofer and Stein, 2011].

We implemented our method as part of the JaTeCs [Esuli et al., 2017] framework using SVMs as the learning device. In all experiments we set the number of pivots to 100. To emulate the word oracle that translates a source pivot word to the target language we used the bilingual dictionaries created by [Prettenhofer and Stein, 2011].

Table 2 and 3 report the results for cross-domain adaptation and cross-lingual adaptation, respectively (bold indicates the best score for each dataset, while greyed-out cells indicate the DCI variants which outperform all other competitors). We refer the interested reader to our original paper [Moreo Fern´andez et al., 2016], where other scenarios (such as simultaneously tackling cross-domain and cross-lingual) are explored.

Most variants of DCI perform comparably or better to all compared methods, even to the new approaches that have been published after our original paper. In this regard, it is interesting to note that many configurations of DCI outperform on average all the competitors. Particularly, the Cosine and Polynomial DCFs have shown to stand the test of time, delivering the best averaged results both in cross-domain and cross-lingual experiments.

5 Conclusions and FutureWork

Distributional Correspondence Indexing is an efficient method for domain adaptation that represents terms in a vectorial space based on their distributional correspondence with respect to a small, fixed set of terms. This representation is motivated by the distributional hypothesis [Harris, 1954] and the notion of a "pivot term" [Blitzer et al., 2006]; the method indexes documents from different domains into a common vector space based on their semantic correspondence. Unlike other distributional semantic methods, DCI gives a lighter interpretation to the hypothesis through the distributional correspondence functions, resulting in a computationally cheap approach to domain adaptation.

Empirical evaluation we have carried out on two popular sentiment analysis benchmarks shows that our method outperforms several state-of-the-art approaches in different domain adaptation settings, including cross-domain and crosslingual sentiment adaptation. DCI has remained unbeaten since its appearance in 2016.

In future research, we plan to put to test DCI in other domains and settings, including multi-class multi- and singlelabel datasets, highly imbalanced classes, and transductive problems.

4.1 Embeddings

The intuitions behind DCI bear strong resemblance to those behind "word embeddings" [Mikolov et al., 2013], as from deep learning. Although neural models typically require large quantities of text data and expensive resources in terms of computation, DCI delivers meaningful representations in a fraction of the time they require (see [Moreo Fern´andez et al., 2016] for a more detailed discussion on efficiency).

Table 4 illustrates the semantic properties captured by our term profiles; it lists the most similar (via cosine similarity) target terms to a given source term.