Feature Ensemble Plus Sample Selection: Domain Adaptation for Sentiment Classification (Extended Abstract)

1 Introduction

The problem of domain adaptation has attracted increasing attention in the fields of both machine learning and natural language processing (NLP). Domain adaptation arises often in sentiment classification [Cambria et al., 2014]. For example, we want to build a book review classifier, in case that the labeled book reviews in hand are scarce but the labeled movie reviews are abundant. Therefore, we need to adapt a classifier trained by labeled reviews in the movie domain to the book domain. Researchers have proposed a variety of domain adaptation approaches in the literatures. In general, the goal of domain adaptation is to build a machine learning model that maximizes the target-domain joint likelihood

∗This paper is an extended abstract of the journal publication [Xia et al., 2013b].

†Email: rxia@njust.edu.cn

based on the source-domain labeled data:

where p(x, y|θ) denotes the joint distribution of instance x and class label y defined in the learning algorithm. ˜ ps(x) and ˜ ps(y|x) denote the approximated target-domain distribution, which should be estimated by the source-domain labeled data.

According to the analysis in [Jiang and Zhai, 2007], there are two distinct needs in domain adaptation: instance adaptation and labeling adaptation. We interpret the two needs as follows:

• Labeling adaptation models the adaptation process of the labeling function ˜ ps(y|x) → pt(y|x). Since one term that is positive in the source domain might express an opposite sentiment in the target domain, labeling adaptation aims to learn a new labeling function (or feature representation) for the target domain, using source domain-labeled data as well as a small amount of labeled (or a large amount of unlabeled) target-domain data;
• Instance adaptation models the adaptation process of instance distribution ˜ ps(x) → pt(x). Since different domains have different term frequencies, instance adaptation aims to approximate the target-domain distribution by assigning different weights to the source-domain labeled data and then conducting importance sampling.

[Pan and Yang, 2010] presented a survey on transfer learning, which categorizes transfer learning approaches in a similar manner: feature-based transfer, instance-based transfer, and model-parameter-based transfer.

Existing work for domain adaptation in sentiment classification mostly belongs to labeling adaptation. [Blitzer et al., 2007] proposed the structural correspondence learning (SCL) algorithm. [Daum´e III, 2007] introduced a simple method for domain adaptation based on feature space augmentation.

Pan proposed two domain adaptation approaches via transfer component analysis (TCA) [Pan et al., 2010] and spectral feature alignment (SFA) [Pan et al., 2011] , respectively. [Xia and Zong, 2011] and [Samdani and Yih, 2011] proposed feature reweighing based on an ensemble of feature sets, although they divide the feature sets in different ways. [Cambria and Hussain, 2015] employed PCA and linear discriminant analysis to build SenticNet, a concept-level resource for open-domain sentiment analysis. They also used affective knowledge to adapt general-purpose common-sense knowledge to the task of polarity detection [Cambria et al., 2015].

The instance adaptation problem is also known as sample selection bias in the machine learning scenario. To address this problem, [Sugiyama et al., 2008] proposed a KullbackLeibler importance estimation procedure (KLIEP) algorithm.

In the field of NLP, [Axelrod et al., 2011] proposed a method called pseudo in-domain data selection to select source domain training samples based on a language model, for cross-domain machine translation. [Xia et al., 2013a] proposed an instance selection and instance weighting approach via PU learning (PUIS and PUIW) for the task of crossdomain sentiment classification. However, current researches in domain adaptation focused on either labeling adaptation or instance adaptation, individually. To the best of our knowledge, research on modeling two kinds of adaptation together is pretty scarce. In this work, we propose a joint method, called feature ensemble plus sample selection (SS-FE), to take full account of these two attributes for domain adaptation in sentiment classification. This approach could yield significant improvements compared to individual feature ensemble (FE) or sample selection (SS) methods, because it comprehensively considers both labeling adaptation and instance adaptation. For more details of this work, see [Xia et al., 2013b].

2 The Proposed Approach

2.1 Feature Ensemble

In formulating our SS-FE method, we first propose a labeling adaptation method via POS-based feature ensemble (FE). This idea is based on the observation that, features with different type of POS tags have a distinct change in distribution in domain adaptation. We term the heavily changing features domain-specific, and the slightly changing features domain-independent. The domain-independent features generally perform more consistently when the domain changes. E.g., some adjectives and adverbs, such as "great" and "like," always have a strong correlation with the positive class label, regardless of domain. However, the domain-specific features may indicate different sentiment in different domains [Cambria et al., 2013], e.g., in the concept "go read the book," the noun "book" most likely indicates a positive sentiment for book reviews, a but negative sentiment for movie reviews.

POS tags are supposed to be significant indicators of sentiment. Previous work revealed a high correlation between the presence of adjectives and document sentiment; certain verbs and nouns are also strong indicators of sentiment [Liu, 2012]. For cross-domain sentiment classification, we observe that features with different types of POS tags might have different levels of distributional change in domain adaptation. For example, nouns change the most because domains are mostly indicated by nouns, while adjectives and adverbs are fairly consistent across domains. The cross-domain Kullback-Leibler (K-L) distance regarding different POS tags (reported in the section of Experiments) confirms our observation.

Based on this observation, we divide features into four groups: adjectives and adverbs (J), verbs (V), nouns (N), and the others (O). Base-classifiers will be trained on all four feature subsets.Thus, a feature vector x is made up of four parts: x = [xJ, xV , xN, xO], and we use gk(xk) to denote the labeling function of each base-classifier:

After base classification, we use the Stacking algorithm for meta-learning, where we construct the meta-learning feature vector ˜x = [gJ, gV , gN, gO] on a small amount of labeled data from the target domain (the validation set), and the weights of base-classifiers are optimized by minimizing the perceptron loss so that each components final weights are tuned to adapt to the target domain. We represent the weighted ensemble as

where θk is the ensemble weight. In the meta-learning process, we expect the algorithm to assign larger weights to the domain-independent parts, such as adjectives and adverbs, and lower weights to the domain-specific parts, such as nouns.

2.2 Sample Selection

The FE approach adapts the labeling function ˜ ps(y|x) → pt(y|x) (in our approach, g → f). However, the labeling adaptation is based on the condition that the instance probability ps(x) and pt(x) are the same. If there is a big gap between them, the effect of labeling adaptation will be reduced. To address this issue, we propose PCA-SS as an aid to FE. PCA-SS first selects a subset of the source domain labeled data whose instance distribution is close to the target domain, and then uses these selected samples as training data in labeling adaptation. Let Xt denote the document-byfeature matrix of the target domain dataset; we then get the target-domains latent concepts by solving the following SVD problem:

where U and V are unit orthogonal matrices and Σ contains the positive singular values of decreasing magnitude σ1 ≥ σ2 ≥ ... ≥ σM. The latent concepts are the orthogonal column vectors in the matrix V, and the variance of the data projected along the i-th column of V is equal to σ2 i .

To optimally capture the datas variations, only those latent concepts corresponding to the k(k < M) largest singular values are typically retained. By selecting the columns of P = V[:, 0 : k] ∈ RM∗k , which correspond to the latent concepts associated with the first k singular values as the projection matrix, we obtain the document-by-concept matrix

Note that Hotelling's T 2 statistic reflects the degree of the magnitude deviation of each sample in a PCA model. In the settings of domain adaptation, it measures the extent to which a sample deviates from the concept space. Therefore, as the criterion for sample selection, we consider the T 2 statistic as a measure of "concept distance":

where z = Λ- 1 2 k PT xT , and Λk is the diagonal matrix corresponding to the top k singular values.

By this definition, we can obtain a concept distance for each sample x(n) s in the source domain:

Correspondingly, we get a set of the concept distances for each sample x(n) t in the target domain: D(x(n) t ) = x(n) t PΛ-1 k PT x(n) t T . Finally, we define the sample selection threshold as:

Samples in the source domain x(n) s with D(x(n) s ) > ¯D are discarded, and those with lower concept distance than the threshold are selected as training samples in domain adaptation.

2.3 Feature Ensemble Plus Sample Selection

In the above-mentioned techniques, FE models only labeling adaptation and neglects instance adaptation, while PCA-SS considers only instance adaptation. Therefore, we combine PCA-SS and FE in a serial manner, and refer to the joint approach as SS-FE, to address two types of adaptation together.

we first employ PCA-SS to project the data onto a latent target-domain concept space, select a subset of sourcedomain samples that are close to the target domain. We then apply FE to the selected part of data set. In FE, we first train individual classifiers with different feature sets divided by their POS tags. The final model is a weighted ensemble of individual classifiers, in which the weights are turned with the goal of increasing the weight of domain-independent features and reducing the weight of domain-specific features.

We empirically show that both FE and PCA-SS are effective for cross-domain sentiment classification, and that SS-FE performs better than either approach because it comprehensively considers both labeling and instance adaptation.

3 Experiments

3.1 Datasets and Experimental Setup

We use the multi-domain sentiment datasets [Daum´e III, 2007] for experiments. It consists of product reviews collected from four different domains of Amazon.com: book (B), DVD (D), electronics (E), and kitchen (K). Each domain contains 1,000 positive and 1,000 negative reviews. The term "source → target" is used to denote different cross-domain tasks. For example, "D → B" represents the task that is trained in the DVD domain but tested in the book domain.

In each of the tasks, labeled instances in the source domain are used for training base-classifiers. Labeled instances in the target domain are evenly split into 10 folds, where one fold is used as a validation set for meta-learning, and the other nine folds are used as a test set. All of the following experimental results are reported in terms of an average of the 10 folds' cross validation.

3.2 Experimental Results of PCA-SS

We first present the results of sample selection. The number of principal components is chosen with the percentage of variance contribution larger than 99.5 percent. All samples in the source and target domains are projected onto the concept space. We sort all source domain samples in an ascending order, according to their concept distance, and present the classification accuracy trained with a decreasing number of selected features in Figure 1. We chose na¨ıve Bayes (NB) as the base classification algorithm, as it reportedly performs the best among three classifiers (NB, MaxEnt, and SVM) in the multi-domain dataset [Xia et al., 2011]. For comparison, we also report the performance of the same number of randomly selected samples. We observe the experimental results from the following perspectives.

Random selection. We first observe the result of randomly selected samples. In all tasks, performance is gradually decreased by reducing the number of selected samples. This is in accordance with our standard understanding that decreasing training samples hurts machine learning performance.

PCA-based sample selection. In most of the tasks, the performance increases when the number of selected samples decreases at the first stage. After reaching the peak value, the performance gradually decreases. This leads to the conclusion that using a selected subset of samples as training data may improve classification performance over training on all samples. The conclusion holds well in the tasks of D → B, B → D, B → E, D → E, B → K, D → K. It is also worth noticing that sample selection is not quite effective in some tasks (such as E → B, K → D, and K → E). We will discuss why later.

3.3 Experimental Results of SS-FE

In this section, we present the results of SS-FE. Four base NB classifiers are trained on the four feature sets, where features with term frequency no less than three are selected and the BOOL weight is adopted. Table 1 reports the classification accuracy of each component classifier (Base-J, Base-V, Base-N, and Base-O), and the system using all features (Unigrams), only FE, only PCA-SS, and SS-FE. We compared the following three aspects.

FE vs. Baselines. We first compare base classifiers and Unigrams. It is interesting that Base-J yields a comparative performance to Unigrams. In some tasks, Base-J is even better. With an efficient ensemble of all base classifiers, FE performs consistently better than each base classifier and the Unigrams system.

PCA-SS vs. Baselines. When comparing PCA-SS and Unigrams, we reconfirm the conclusion from Figure 1 that, with a selected subset of training samples, the PCA-SS could improve significantly.

FE, PCA-SS, and SS-FE. SS-FE is consistently better than Unigrams, FE, and PCA-SS, except for one task K → B, where SS-FE is slightly weaker than FE. We summarize the results as follows: first, in the tasks, such as D→ B and K → D, where FE is more effective but the PCA-SS is less effective (denoted as FE ≻ SS), the SS-FE improvements are generally gained by labeling adaptation. Second, in the tasks where the effects of PCA-SS are more significant than FE (FE ≺ SS), such as E →D and E →K, the SS-FE improvements are mainly from instance adaptation. Third, in the tasks where FE and PCA-SS are both effective, such as D → E and B → K, the improvements finally gained by SS-FE are remarkable.

3.4 Why POS-Based Feature Ensemble?

Table 2 shows the average KL distance (KLD) across all tasks regarding to different POS tags and their weights learnt in FE. Generally, the KLD of different types of POS tags can be ranked as: N >>V > J > O. The KLD of N is significantly larger than the other POS tags, indicating that the change of N is the biggest across domains. The KLD of J is significantly smaller than that of N. V gives the comparable KLD. This suggests that features in J and V are more domain-independent, in comparison with N. We then observe the weights of different parts of POS tags learnt in FE. The weight of J is the largest, while the weight of N is comparatively small. This confirms our motivation for POS-based feature ensemble.

3.5 Why Feature Ensemble plus Sample Selection?

In this work, labeling adaptation is conducted in a feature reweighing manner by FE. Instance adaptation is embodied in the manner of sample selection, where the empirical likelihood is obtained based on the selected subset of samples. To validate this analysis, we further present the KLD of different domain pairs in Table 3. We find that the KLD of B-D and E-K are relatively low, which suggests that these domains are similar in distribution. This corresponds well with our experimental results: first, in tasks such as D → B and K → D, the improvement of SS-FE is generally gained by labeling adaptation. This is quite reasonable, because their KLD is relatively low and the demand for instance adaptation is not large. Second, for the domain pairs whose distributional change is bigger, such as E → D and E → K, the improvement of SS-FE is mainly due to instance adaptation rather than labeling adaptation.

4 Conclusions

We have presented the SS-FE approach to conduct labeling adaptation and instance adaptation together for domain adaptation. Experimental results showed the effectiveness of SSFE in both labeling adaptation and instance adaptation.

Acknowledgement

The work is supported by the Natural Science Foundation of China (61305090), the Jiangsu Provincial Natural Science Foundation of China (BK2012396), and the Research Fund for the Doctoral Program of Higher Education of China (20123219120025).