Concept Learning for Cross-Domain Text Classification: A General Probabilistic Framework

1 Introduction

Traditional machine learning algorithms often fail to obtain satisfactory performance when the training and test data are drawn from different but related data distributions. See the task example as follows. To build an enterprize news portal we need to classify the news about a certain company into some predefined categories, such as "product announcement", "financial analysis", "employee laid-off" and so on. This classification model may be trained from the news about one company, and may fail on the news for another company since the business areas for the two companies may be different. Thus, cross-domain learning (referred as domain adaptation or transfer learning), focusing on adapting the models trained from source domains to target domains, attracts many research interests recently [Pan and Yang, 2010; Dai et al., 2008; Gao et al., 2008; Dai et al., 2007a; Luo et al., 2008; Xue et al., 2008; Dai et al., 2009; Zhuang et al., 2010b; 2010a; Wang et al., 2011; Long et al., 2012]. Some of these previous works have showed that the high-level word concepts are more appropriate for across-domain text classification instead of the raw word features. Specifically, they consider two sides of a concept, namely concept extension and intension.

Definition 1 (Extension of Word Concept) The extension of a word concept z is represented by a multinomial distribution p(w|z) over words.

It is actually the degree of applicability of each word w for that concept z. That is to say, when p(w|z) is large, w is a word to the word concept z.

Definition 2 (Intension of Word Concept) The intension of a word concept z is expressed by its association with each document class y, denoted by the conditional probability p(z|y) in this study.

It indicates that when p(z|y) is large, the concept z is strongly related to the document class y.

Zhuang et al. [Zhuang et al., 2010b; 2010a] showed that the extension of a concept is often domain-dependent. See the concept of "product" in the task of enterprize news classification. If this concept is in the domain of HP ( which makes printers), the values of p("printer"|"products",HP) and p("LaserJet"|"products",HP) are large. However, these two values p("printer"|"products", IBM) and p("LaserJet"|"products", IBM) will be very small in the domain of IBM, since IBM seldom sells printers. Additionally, their work observed that the intension of a concept, namely p(z|y), is usually stable across domains. In other words, wherever a word concept exists, it has the same implication to the class of the document. For example, if a news contains the word concept "products", no matter where it comes from, it is more likely to be a news about "product announcement" rather than about "employee laid-off". Based on these observations we give the definition of homogeneous concepts as follows.

Definition 3 (Homogeneous Concepts) A concept zb is homogeneous if it has the same intension but different extensions across the data domains.

In other words, since the intension of a homogeneous concept zb remains the same across data domains we have

where r, r′ are two of any data domains. Thus, to represent the intension of a homogeneous concept we do not need the variable r of data domain in the conditional probability. Thus, it can be represented as p(zb |y). However, since the extension of a homogeneous concept changes in different data domains we have

Thus, the extension of a homogeneous concept can only be represented as p(w|zb , r), where the variable r of data domain is included in the condition.

Besides the homogeneous concepts, the data domains may also share some concepts with the same intension and extension. For example, the enterprize news about "financial analysis" may talk about the concept of "accounting terminologies". Different companies may use the same set of standard terminologies. Thus, the intension of this concept remains the same even in different domains. Thus, we can give the definition of identical concepts as follows.

Definition 4 (Identical Concepts) A concept za is identical if it has the same intension and the same extension across the data domains.

Similarly, for an identical concept za we have

where r, r′ are two of any data domains. Therefore, the intension and extension of an identical concept can be represented as p(za |y), p(w|za ) respectively, where the variable r of data domain can be omitted.

It is clear that both homogeneous and identical concepts are shared among all the data domains. In this study we further argue that some concepts may only exist in one of the data domains. See the task of enterprize news classification again. Assume that one of the companies is in the process of stock market launch. Thus, the news about this company may mention the concept about the "IPO process". Since all the other companies have been listed for a long time, the concept about "IPO process" may only exist in the news about this company. With this observation we can give the definition of distinct concepts as follows.

Definition 5 (Distinct Concepts) A concept zc is distinct if it only exists in one of the data domains.

If a distinct concept zc exists only in the data domain r, its intension and extension can be represented as p(zc |y, r), p(w|zc , r) respectively. However, for any other domain r′ the values of p(zc |y, r′ ), p(w|zc , r′ ) are invalid.

In summary we observe three kinds of concepts in the data domains for transfer learning, as shown in Figure 1. The homogeneous and identical concepts are shared among all the data domains, while the distinct concepts may only exist in one of them. Thus, we need a general model to capture all these concepts in all the data domains simultaneously. Hopefully, it may lead to better classification performance with the increase of the model expression ability. Along this line, we propose a Homogeneous-Identical-Distinct-Concept (HIDC for short) model based on the probabilistic methods. Then, an EM algorithm is developed to solve the likelihood maximum problem. Finally, we conduct systemic experiments to show the effectiveness of the proposed model as well as its superiority over the compared baselines.

2 Preliminary Knowledge and Problem Formulation

In this section, we first briefly review Probabilistic Latent Semantic Analysis (PLSA) [Hofmann, 1999], and then introduce an extension of PLSA, Dual-PLSA [Jiho and Choi, 2009]. Finally, we formulate the optimization problem for cross-domain text classification which incorporates all the three kinds of concepts.

2.1 Preliminary on PLSA and D-PLSA

Probabilistic Latent Semantic Analysis [Hofmann, 1999] is a statistical model to analyze co-occurrence data by a mixture decomposition. Specifically, given the word-document co-occurrence matrix O whose element Ow,d represents the frequency of word w appearing in document d, PLSA models O by using a mixture model with latent topics (each topic is denoted by z) as in Eq.(1).

Figure 2(a) shows the graphical model for PLSA. The parameters of p(w|z), p(d|z), p(z) over all w, d, z are obtained by the EM solution to the maximum likelihood problem.

In the PLSA model, the documents and words share the same latent variable z. However, documents and words usually exhibit different organizations and structures. Specifically, they may have different kinds of latent topics, denoted by z for word concept and y for document cluster. Its graphical model is shown in Figure 2(b). Since there are two latent variables in this model we call it Dual-PLSA (D-PLSA for short) [Jiho and Choi, 2009] in this paper.

Given the word-document co-occurrence O, we can similarly arise a mixture model like Eq.(1),

And the parameters of p(w|z), p(d|y), p(z|y), p(y) over all w, d, z, y can also be obtained by the EM solution.

2.2 The Graphical Models for Identical, Homogeneous, and Distinct Concepts

Motivated by D-PLSA, we propose three probabilistic generative models in Figure 3, which correspond to the three kinds of concepts in Definitions 3 through 5. In these sub-figures, w and d refer to any word and any document respectively, z and y refer to any word concept and any document cluster respectively, and r refers to any data domain. Additionally, the word concept z can be further divided into three kinds, namely za , zb , and zc , representing identical concepts, homogeneous concepts, and distinct concepts respectively.

Figure 3: The Graphical Models

First, Figure 3(a) considers the identical concepts, where both the concept extension p(w|za ) and intension p(za |y) are irrelevant to the data domain r. It indicates that no matter where za comes from it has the same extension and intension. Second, Figure 3(b) considers the homogeneous concepts. You can see that in Figure 3(b) the concept extension p(w|zb , r) is dependent on r while the intension p(zb |y) is irrelevant to r. It indicates that any homogeneous concept zb has the stable intension but different extensions across the domains. Finally, Figure 3(c) considers the distinct concepts. Here, since any distinct concept zc exists only in r, its extension p(w|zc , r) and intension p(zc |y, r) are both relevant to the data domain r.

Given the graphical models in Figure 3, the joint probability over all the variables is

Here, the concept z is expanded by za , zb , zc . Additionally, based on the graphical models in Figures 3(a), 3(b) and 3(c), we have

2.3 ProblemFormulation

Suppose we have s + t data domains, denoted as X = (X1, · · · , Xs, Xs+1, · · · , Xs+t). Without loss of generality, we assume the first s domains are source domains with label information, i.e., Xr = {x(r) i , y(r) i }|nr i=1 (1 ≤ r ≤ s), and the left t domains are target domains without any label information, i.e., Xr = {x(r) i }|nr i=1 (s + 1 ≤ r ≤ s + t), nr is the number of documents in data domain Xr.

We try to maximize the log likelihood in Eq.(7) as follows,

where Z includes all the latent variables, and θ represents all the model parameters.

In the following section, we develop an EM algorithm to solve the optimization problem in Eq.(7).

3 Solution to HIDC

An Expectation-Maximization (EM) algorithm is to maximize the lower bound (via Jensen's inequality) L0 of Eq.(7):

where q(Z) could be arbitrary probability distribution over all latent variables. For computation convenience, we set q(Z) = p(Z|X; θold ), and omit the constant. Then the objective function L0 becomes L,

Here, we denote za , y as Za , zb , y as Zb , zc , y as Zc .

3.1 E-step

Suppose Xr includes the variables w, d, r, and Or corresponds to word-document co-occurrence of data Xr (1 ≤ r ≤ s+t), we can obtain Eq.(10) by substituting all variables into Eq.(9),

where Ow,d,r represents the co-occurrence frequency of the triple (w, d, r). Note that p(za , y,w, d, r), p(zb , y,w, d, r) and p(zc , y, w, d, r) can be respectively calculated by Eqs.(4), (5) and (6). Therefore, the iterative formulas in E-step are as follows,

3.2 M-step: maximizing L

Now we maximize L with its parameters by Lagrangian Multiplier method. Take the derivation of p(d|y, r) as an example, and extract the terms containing p(d|y, r). Then, we have

Applying the constraint ∑d p(d|y, r) = 1 into the following equation:

then

3.3 HIDC to Cross-domain Classification

In this subsection, we introduce how to leverage the proposed EM algorithm for cross-domain text classification. There are two sub-tasks: 1) how to inject the label information in source domains to supervise the EM optimization; 2) how to assign the class label to the instances in the target domains based on the output from the EM algorithm.

Algorithm 1 HIDC for Cross-domain Text Classification

Input: Given (s + t) data domains X1, · · · , Xs, Xs+1, · · · , Xs+t, where the first s domains are source domains with label information, while the left are target domains. T, the number of iterations. k1, k2, k3, the number of identical concepts, homogeneous concepts and distinct concepts. (In this study we simply assume each domain has the same number of distinct concepts.) Output: the class label of each document d in the target domains.

Considering the constraint ∑d p(d|y, r) = 1,

1. Initialization. The detailed initialization of p(0) (w|za ), p(0) (w|zb , r), p(0) (w|zc , r) can be referred in Section 4.2. The initialization of p(0) (d|y, r) is detailed in Section 3.3. p(0) (za |y), p(0) (zb |y), p(0) (zc |y, r), p(y|r) and p(r) are set randomly.

Finally, the update formula of p(d|y, r) can be obtained,

Similarly,

1. end.
2. Update p(k) (w|za ), p(k) (za |y) and p(k) (zb |y) according to Eqs.(19), (22) and (23) inM-step;
3. for r := s + 1 → s + t Update p(k) (d|y, r) according to Eq.(18) in M-step;
4. end.
5. k := k + 1, if k < T, turn to Step 3.
6. The class label of any document d in a target domain is predicted by Eqs.(27) and (28).

For the first task we inject the supervising information (the class label of the instances in the source domains) into the probability p(d|y, r) (1 ≤ r ≤ s). Specifically, let Lr ∈ {0, 1}nr×m be the true label information of the r-th domain, where nr is the number of instances in it, m is the number of document classes. If instance d belongs to document class y0, then Lrd,y0 = 1, otherwise Lrd,y = 0 (y ̸= y0). We normalize Lr to satisfy the probability condition so that the sum of the entries in each column equals to 1, N r d,y = Lrd,y ∑d Lrd,y . Then p(d|y, r) is initialized as N r d,y . Note that since this initial value is from the true class label we do not change the value of p(d|y, r) (for 1 ≤ r ≤ s) during the iterative process.

For the unlabeled target domains, p(d|y, r) (s + 1 ≤ r ≤ s+t) can be initialized similarly. This time the label information Lr can be obtained by any supervised classifier (Logistic Regression is used in this paper). Note that since this classifier may output the wrong class label we do change the value of p(d|y, r) (for s + 1 ≤ r ≤ s + t) during the iterative process.

After the EM iteration we obtain all the parameters of p(d|y, r), p(w|za ), p(w|zb , r), p(w|zc , r), p(za |y), p(zb |y), p(zc |y, r), p(y|r) p(r), based on which we compute the posteriori probability p(y|d, r) as follows,

Then, the class label of any document d in a target domain r is predicted to be

The detailed procedure of HIDC for cross-domain text classification is depicted in Algorithm 1.

4 Experimental Evaluation

In this study, we focus on evaluating our model on binary classification scenarios. Note that our model can naturally handle multi-class classification tasks.

4.1 Data Preparation

20Newsgroups 1 is one of the mostly used data sets for evaluating transfer learning algorithms [Pan and Yang, 2010; Gao et al., 2008; Dai et al., 2007b; Zhuang et al., 2010a]. This corpus has approximately 20,000 newsgroup documents, which are evenly divided into 20 subcategories. Some similar subcategories are grouped into a top category, e.g., the four subcategories sci.crypt, sci.electronics, sci.med and sci.space belong to the top category sci. We use four top categories comp, rec, sci and talk to construct classification tasks, and each top category contains four subcategories.

To validate the effectiveness of HIDC, we construct the transfer learning problems as follows. Firstly, we choose two top categories rec and sci, one as positive class and the other one as negative one. To form a source domain, we randomly select one subcategory from rec and another subcategory from sci. The target domain is similarly constructed. Thus in this way we can obtain 144 (P 2 4 ×P 2 4 ) tasks. In these tasks since the documents in the source and target domains are drawn from the same two top categories, and they may share lots of homogeneous and identical concepts, but have less distinct concepts. Thus, we call this kind of tasks as tasks with less distinct concepts.

Secondly, to intentionally construct the classification scenario when the distinct concepts may exist, we replace one subcategory for the target domain with another subcategory from the top category comp or talk. For example, the source domain has the documents from the top categories of rec and sci, while the target domain has the ones from rec and talk. Then, totally 384 (P 2 4 ×P 1 4 ×8) problems can be constructed. In these tasks since the documents in the target domain may come from the top category which is not included in the source domain, there may exist lots of distinct concepts in them. Thus, these new transfer learning tasks might be much more challenging. To avoid negative transfer, among all the 384 tasks we only consider the 334 tasks on which the supervised learning model Logistic Regression (LR) [Hosmer and Lemeshow, 2000] outputs the accuracy higher than 50%. We call this kind of tasks as tasks with more distinct concepts.

1 http://people.csail.mit.edu/jrennie/20Newsgroups/.

In summary, we have 144 traditional transfer learning tasks and 334 new ones from the 20Newsgroups data set.

Sentiment Data. We use the Multi-Domain sentiment data set 2 [Blitzer et al., 2007], which contains product reviews taken from Amazon.com for many product types (domains) to further demonstrate the effectiveness of HIDC. This data set contains the positive and negative reviews from four domains, i.e., books, dvd, electronics and kitchen.

Here, to show that HIDC can naturally deal with the classification scenarios which have multiple source and target domains, we construct the tasks with two source domains and two target domains from the Sentiment Data. Specifically, we randomly select two domains as source domains, and the rest two as target domains. Thus we have 6 classification problems.

4.2 Experimental Setup

Compared algorithms: We compare our model HIDC with some state-of-the-art baselines, including 1) the supervised algorithms: Logistic Regression (LR) [Hosmer and Lemeshow, 2000], Support Vector Machine (SVM) [Boser et al., 1992]; 2) the semi-supervised algorithm: Transductive Support Vector Machine (TSVM) [Joachims, 1999]; 3) the cross-domain methods: Co-clustering based Classification (CoCC) [Dai et al., 2007a], CD-PLSA [Zhuang et al., 2010a] and Dual Transfer Learning (DTL) [Long et al., 2012].

Parameter setting: In Algorithm 1, we set k1 = 20, k2 = 20, k3 = 10 and T = 100 (Additional experiments show that HIDC is not sensitive to the parameter setting when they are sampled from some predefined bounds, and converges very fast. Due to the space limitation, we don't detail them here.). The parameters p(w|za ), p(w|zb , r) and p(w|zc , r) (1 ≤ r ≤ s + t) are initialized as follows. We combine all the data from source and target domains, and conduct the PLSA implemented by Matlab 3 . We set the number of topics as (k1 + k2), and obtain the word clustering information W ∈ Rm×(k1+k2) + (m is the number of word features). W is divided into two parts W = [W 1 ,W 2 ] (W 1 ∈ Rm×k1 + , W 2 ∈ Rm×k2 + ), then p(w|za ) is initialized as W 1 and p(w|zb , r) is assigned as W 2 . Finally, p(w|zc , r) is randomly initialized, while ensuring ∑w p(w|zc , r) = 1. Due to the random initialization, we run ten times for each problem and report the average performance. As for the baseline methods, LR is implemented by Matlab 4 , SVM and TSVM are given by SVMlight5 . The parameters of CoCC, CD-PLSA and DTL are set as the default ones in their original papers, since their authors have demonstrated they can perform well on the 20Newsgroups data set. We use the classification accuracy as the evaluation metric.

2 http://www.cs.jhu.edu/ mdredze/datasets/sentiment/.

3 http://www.kyb.tuebingen.mpg.de/bs/people/pgehler/code/.

4 http://research.microsoft.com/∼minka/papers/logreg/.

5 http://svmlight.joachims.org/.

4.3 Experimental Results

Results on the 20NewsGroup Data

For the 334 tasks with more distinct concepts we further divide them into two parts, i.e., 65 tasks whose accuracies from LR are lower than 55%, and the other 269 ones higher than 55%. The average accuracies on these tasks for all the compared algorithms are listed in Table 1.

When the accuracies from LR are lower than 55%, we can find that 1) these problems are very difficult, since both of the supervised machine learning algorithms LR and SVM perform only slightly better than random classification. 2) HIDC achieves the outstanding results, which is significantly better than all the other algorithms. Especially, HIDC can obtain 37.67% average accuracy improvement compared with LR. 3) HIDC performs better than TSVM, since TSVM still assumes the source and target domains are drawn from the same data distributions. 4) CD-PLSA is better than DTL, and DTL is better than CoCC. In summary these results show that modeling the distinct concepts in HIDC helps to significantly improve the classification performance.

When the accuracies from LR are higher than 55%, we can still see that all the four cross-domain algorithms perform better than the traditional supervised algorithms, and HIDC still outperforms all other ones. We can also see that the performance improvement from HIDC under this situation is less than that for the tasks whose LR accuracies are less than 55%.

Similarly, we can also divide the 144 tasks with less distinct concepts into two parts, i.e., with the LR accuracy higher or lower than 55%. As shown in Table 1, again HIDC beats all the baselines.

Results on Sentiment Data

Note that the tasks on Sentiment Data have the two source domains and two target domains. For the supervised algorithm LR [Hosmer and Lemeshow, 2000], we record the ensemble performance of the classifiers from the two source domains. Also, we compare HIDC with the cross-domain algorithms DTL [Long et al., 2012] and CD-PLSA [Zhuang et al.,

2010a], both of which can handle multiple source domains and multiple target domains directly.

The experimental results are shown in Table 2. For each task we record the accuracy values of the two target domains respectively. Again, HIDC exhibits the best performance.

5 RelatedWork

Transfer Learning arouses vast amount of studies in recently years [Pan and Yang, 2010], and we briefly introduce the most recent and related works to this paper.

The work of transfer learning can be approximately grouped into three categories, namely instance-based approaches [Dai et al., 2007b; Jiang and Zhai, 2007], weighting-based methods [Gao et al., 2008; Dredze et al., 2010], and feature-based algorithms [Dai et al., 2007a; Long et al., 2012; Li et al., 2009; Wang et al., 2011; Zhuang et al., 2010a]. Our model HIDC belongs to the feature-based algorithms, thus here we focus on reviewing works of this category. Dai et al. [Dai et al., 2007a] proposed a Co-Clustering based Classification (CoCC) algorithm, and used the common words for knowledge transfer. Essentially, they only considered the identical concepts. Zhuang et al. [Zhuang et al., 2010a] and Wang et al. [Wang et al., 2011] actually exploited the homogeneous concepts for knowledge transfer. The most related work is Long et al. [Long et al., 2012], where they developed a Dual-Transfer Learning (DTL) framework, and took into account both the identical and homogeneous concepts. However, they did not explicitly consider the distinct concepts. Our model HIDC fully exploits the three kinds of concepts, and the experimental results show the superiority over all compared baselines. It is worth mentioning that considering the distinct concepts is very important and not a trivial work, since HIDC significantly outperforms DTL.

6 Conclusions

In this paper, we deeply analyze the characteristics between the source and target domains, and define three kinds of concepts, i.e., homogeneous concepts, identical concepts and distinct concepts. Along this line, we propose a general cross-domain learning framework based on generative models, which takes into account the three kinds of concepts simultaneously. Then an EM algorithm is developed to derive the solution of our model. Finally, we conduct extensive experiments, no matter the distinct concept is intentionally introduced or not, to demonstrate the superiority of our model.

Acknowledgements

The work is supported by the National Natural Science Foundation of China (No.61175052, 61203297, 60933004, 61035003), National High-tech R&D Program of China (863 Program) (No.2013AA01A606, 2012AA011003), National Program on Key Basic Research Project (973 Program) (No.2013CB329502).