Personalized Sentiment Classification Based on Latent Individuality of Microblog Users

1 Introduction

With the popular microblogging services such as Twitter and Sina Weibo, users can conveniently express their personal feelings and opinions about all kinds of issues in real time. The enormous microblog data have fueled us with torrents of subjective text. Mining users' sentiment orientation from such vast mount of subjective text has drawn close attention from research communities in recent years.

Much work was done for detecting sentiment orientation from user-generated content including reviews [Pang and Lee, 2005], online discussions [Hassan et al., 2010], blogs [Feng et al., 2011] and microblogs [Hu et al., 2013]. However, most of studies ignore the personal distinctions among different users providing the subjective text. Typically, social media users exhibit various styles when expressing their feelings online. It is hypothesized that such diversity of sentimental manifestation may be pertinent to the latent aspects of different people including their personality, educational background, current mood and some unknown factors. While using the same wording, people may deliver different sentiment orientations depending on the underlying context, which can be shown by the following example where some users are tweeting about work overtime:

In this example, user A is a fan of body fitness who enjoys losing weight by hard work, and user C is a complainer who grumbles about hard work causing him loss of health. For the second sentences of both users, if not knowing their background, traditional sentiment models can hardly predict correct polarity without looking into user's identity. Given individual vocabulary choices, it is expected that useful latent information reflecting different individuality could be captured and leveraged to determine the user's sentiment orientation more accurately. Also, if we know that B is A's follower in a Yoga class, and D follows C being C's colleague, the latent individuality of target users could be strengthened further based on such relation among users. Thus, it would be more likely that they may share similar interests and hold consistent sentiment orientation with their respective followees.

In this paper, we aim at catching latent personal distinctions or individuality among different writers of subjective text. We propose a novel, extensible and effective Personalized Sentiment Classification method based on a variant of Latent Factor Model (LFM) that realizes sentiment personalization under the sparse distribution of microblog texts which is very common in social media environment. The main contributions of our paper are three-fold:

• We propose a novel, extensible latent-factor-based personalized sentiment classification model, which alleviates the sparsity of training data by decomposing microblog posts into a finer-grained representation via probabilistic matrix factorization and maps users and microblog contents into a shared latent factor space. The latent factors reflect the interested aspects of both users

and contents, which correspond to individuality of users.

• Our sentiment analysis module takes into account different roles of syntactic units (primarily sentiment units and topic units) identified from words' dependency relations, which provide rich emotional and topical clues regarding the interested aspects of different users. Our factorization model achieves more accurate estimation of personalized sentiment scores of the posts by automatically weighing different types of syntactic units.
• We integrate social relation into the model for enhancing the personalized sentiment representation of target users, which is based on the intuition that the following relation between followers and followees can reflect the shared latent individuality of users.

2 RelatedWork

Research for understanding sentiments of microblogs has been an active research area in recent years [Go et al., 2009; Pak and Paroubek, 2010; Davidov et al., 2010; Wang et al., 2011; Tan et al., 2011; Hu et al., 2013; Hutto and Gilbert, 2014]. Go et al. [2009] and Pack and Paroubek [2010] both acquired sentiment data from Twitter for learning sentiment models. Hutto and Gilbert [2014] presented VADER, a simple yet effective rule-based model for general sentiment analysis of social media text. Speriosu et al. [2011] proposed a semi-supervised approach for polarity annotation by using label propagation over lexical links and follower graph. Wang et al. [2011] automatically generated the overall sentiment polarity for a given hashtag relevant to some coarse-grained topics. Tan et al. [2011] assumed that users that are somehow connected may be more likely to hold similar opinions, therefore, they used social relation to complement a user's viewpoints from their utterances. Hu et al. [2013] studied social relations and proposed a sociological approach to handle microblogging texts for sentiment classification. These studies did not deal with personalization. Our goal is to personalize the sentiment model by differentiating latent individuality of microblog users that are not articulated explicitly and dealing with the sparsity of personal microblog data.

Latent factor model, as a matrix factorization method, has been widely used in online recommendation systems. LFM can capture the hidden elements determining users' preferences which are commonly difficult to analyze. Agarwal et al. [2011] proposed a factor model that incorporates rater-comment and rater-author interactions to rank the comments associated with a given article according to user preference. Chen et al. [2012] proposed a collaborative tweet ranking model for recommending tweets to users. Li et al. [2009] worked on a constrained non-negative trifactorization method based on term-document matrix to learn a sentiment model from lexical priori knowledge, but they failed to personalize the model. Different from these works, we incorporate the factors induced from social, sentimental and topical evidence into the matrix factorization process and propose a novel extensible LFM for personalized sentiment modeling on microblogs data.

Personalized sentiment model was rarely explored for microblog users. Calais Guerra et al. [2011] argued that users'

opinion bias always existed in microblogs for some particular entities, and proposed a user bias quality method to transfer user biases into opinion features for specific topics. Li et al. [2011a] proposed a tensor factorization model for review rating, where the reviewer, product and text features were modeled as a three-dimension tensor, to which our work is more closely related. But our work differs significantly from theirs: (1) Their model is particularly customized for product review rating and cannot be directly applied for our microblogging texts independent of specifiable products or objects; (2) Their model is simply based on bag of words which is not as extensible as ours to model fine-grained sentiment-topic expressions and incorporate user connections to strengthen the capturing of useful individuality.

3 Latent Factor Model for Microblog Users

Inspired by the collaborative personalized tweet recommendation [Chen et al., 2012], we try to model the personalized sentiment of microblog users based on LFM. However, we argue that the LFM has poor extensibility for unobserved microblog posts rendering the direct application of it impractical. In this section, we first introduce the basic setting of LFM, then we extend the model by decomposing the post items into word level for alleviating the sparsity of personal microblog data, and also incorporating social relation evidence for personalization modeling. In this section, we give its theoretical derivation, which paves the way for our personalized sentiment model.

3.1 Formalization of LFM

Let U = {u1, u2, . . . , u|U|} be a set of users, and I = {i1, i2, . . . , i|I|} be a set of microblogs posted by U. For each u ∈ U, Iu ⊂ I is the set of posts by u, and |Iu| ≪ |I|. The posts with sentiment labels form the set I ∗ u (I ∗ u ⊂ Iu), and similarly |I ∗ u | ≪ |Iu|. For any post i ∈ I ∗ u , we observe a ground-truth polarity xui (xui = 1 if the observed sentiment of i is positive, or xui = 0 otherwise). Each training instance is represented by a tuple (xui, u, i), which is organized into a sparse matrix X of size |U|×|I|, using (u, i) as index and xui as entry value. The task is to predict the sentiment score ˆxui for the missing items in the set I + u = Iu - I ∗ u . LFM tries to approximate X by the product of two low-rank latent factor matricesW: |U| ×f andH: |I| ×f, where f is a parameter corresponding to the rank of the approximation, and in general f ≪min{|U|, |I|}. Therefore, each observed score xui in X can be approximated by ˆxui in ˆX, which is the product of two components:

where Wu is the user feature vector and Hi is the post feature vector both in f-dimensional space that is referred to as interested aspects of users and contents. For users, interested aspects indicate their latent individuality; in posts, sentiments are expressed towards the interested aspects, which together help model the personalized sentiment scoring for ˆxui.

The direct application of LFM has some critical problems: (1) The extensibility is limited since the model has to be retrained when a new post comes in; (2) Each post is treated as an entire item, thus its consisted components containing crucial personal sentiment features cannot be utilized; (3) The personal data in matrix X is generally very sparse, which makes factorization ineffective.

3.2 Our Extension: Word Level Decomposition and Following Relation Incorporation

To resolve these problems, our idea is to derive an extensible model that decomposes posts into some finer-grained level. In this section, we consider words as the basic unit in the first place without complicating the model too much (In the next section, we will consider coupling words of different functions for further enhancement). Meanwhile, it is intuitive that target users may have shared interested aspects with their followees. Therefore, to enhance personalization, we incorporate users' following relations into the factorization model. Figure 1 illustrates our extension which is detailed below.

Suppose there is a vocabulary K containing all the words in the dataset, and then we decompose H as H = Q·VT , where Q: |I|×|K| is a post-word matrix whose i-th row is the vector of post i indicating the existence of its component words, and V: f × |K| is an estimated word-factor matrix projecting each word k into feature vector VT k of size f. Therefore, the estimated matrix ˆX = W · (Q · VT )T = W · (V · QT ), and formula 1 can be reformulated as:

To incorporate social connections, we project the userfactor matrix W into M+ CM with a newly estimated userfactor matrix M: |U| × f and an observed followee-follower connection matrix C: |U| × |U| where each entry Cuv indicates whether user u follows user v (its value is normalized instead of binary). As a result, formula 2 is extended as:

For estimating ˆX, we derive an optimization objective based on probabilistic matrix factorization [Salakhutdinov et al., 2007]. A linear model with Gaussian observation noise can be adopted to generate the observed variables. We can obtain the probability of generating X as P(X|M,V) = ∏|U | u=1 ∏|I | i=1 [N(xui|ˆxui, δ2 )]1ui , where N(x|µ, δ2 ) is the Gaussian distribution with mean µ = ˆxui and variance δ2 , and 1ui is an indicator function which takes 1 if u and i are observed in the training set or takes 0 otherwise.

Then we can place a zero-mean Gaussian prior distribution on user and word feature vectors as follows: P(M) =

∏|U | u=1 N(Mu|0, δ2U E) and P(V) = ∏|K| k=1 N (Vk|0, δ2K E), where E is an identity matrix, δ2U is the variance of user features, and δ2K is the variance of word features.

Based on Bayesian formula, we can obtain the posterior P(M,V|X) = P(X|M,V)×P(M)×P(V) P (X) . Then the parameters (i.e., features in latent space) Θ = (M,V) of the model can be found by taking the logarithm of the posterior and using maximum a posterior estimation. Finally, we derive the objective function by minimizing the sum of squared errors with a regularization term:

where Regularizer = λU ∑u ‖Mu‖22 + λK ∑k ‖Vk‖22 , λU = δ2 δ2U and λK = δ2 δ2K are the fixed coefficients corresponding to users and words, respectively. The first term strives to fit the given sentiment scores, and the regularizer avoids overfitting by penalizing the magnitudes of the parameters. After estimating the parameters Θ, we can predict the sentiment score of any unobserved post by using formula 3.

4 Personalized Sentiment Modeling

Although the word-level decomposition can make post-word matrix Q denser, the bag-of-words model is still rather coarse for sentiment analysis, which might drag the parameter estimation with respect to words in V. Some basic problems should be considered: (1) Negation often modifies sentiment dramatically, e.g., from positive to negative; (2) It is necessary to differentiate sentiment words and non-sentiment words while some non-sentiment words may need special treatment. For example, intensifiers (e.g., 'extremely', 'very', etc.) are sentiment shifters and should be modeled together with sentiment words they modify; (3) Prior knowledge, e.g., sentiment values in a sentiment lexicon, is important. We can treat (3) straightforwardly by letting the entry of Q take the sentiment value for sentiment word (sentiment values are usually larger than 1).

Other than sentimental evidence, topic-related words may reflect users' individuality which need to be modeled appropriately. In fact, the hidden topics can naturally result from the interested aspects in the factorized matrix V which interact with users via the user-factor matrix. We just need to specify topical words and treat them specially in the model.

To compromise sparsity alleviation and model's effectiveness, we resort to the syntactic units in the posts derived from dependency parsing as intermediary for better estimating the word features in the latent space, which is detailed below.

4.1 Syntactic Units fromDependency Parsing

Words are not independent of each other in a sentence because usually there exists dependency relation among them. Syntactic units based on dependency can capture longdistance word relation and convey finer-grained sentiment and topic information than bag of words, bigrams or trigrams. When estimating word features, we will consider such dependency relation to improve the accuracy of estimation

(see Section 4.2). Here we describe how the syntactic units can be extracted.

Figure 2 shows an example of typed dependencies represented as a directed graph, where a head word points to a dependent word on each edge denoting that the dependent somehow modifies the head. The dependencies are all binary relations, so we can obtain a group of word pairs such as {sent, necklace}, {shinny, very}, {fake, not}, etc., where we do not differentiate word order. While microblog text is typically informal and noisy, we still resort to the Stanford neural network parser 1 , which is trained on formal text, to parse the posts. However, we lexically normalize the posts and correct ill-formed words during preprocessing by referring to [Han et al., 2013]. This saves the intensive labor cost to annotate data for training a tailored microblog parser while we can still obtain reasonable parsing result.

One issue is that the obtained pairwise units from dependency parsing are not all useful. Therefore, we resort to sentiment lexicon and part-of-speech (POS) of the words to extract those units that provide sentimental and topical evidence, referred to as sentiment units and topic units, respectively. We adopt the following rules for extraction:

1. Units containing words in sentiment lexicon are kept as sentiment units; Other units which contain noun or verb are kept as topic units because typically nouns and verbs can describe the topics concerned;
2. Adverbs in the units obtained after rule 1 are kept because they are indispensible in expressing the intensity (e.g., very) or shift (e.g., not) of emotions, or the accomplishment of some actions (e.g., already);
3. Words with other types of POS are removed from the units which are generally not sentimental nor topical;
4. Due to the removal of some words from the pairs after rule 3, it can result in singletons. A resulted singleton is either just left as it is or merged into larger units that contain that word.

4.2 Integrating Sentiment Units and Topic Units

Given the syntactic units extracted, we need to specify the entries' values in Q for better estimating the word feature values in V. Note that we use words rather than the identified syntactic units as the ultimate representation in matrices because the chance of having unseen words is generally lower than having unseen units in test data. But we will consider the extracted word relation for parameter estimation.

Under bag-of-words setting, the entry of Q in formula 2 can take 1 |Ki| for the observed words in post i and 0 for the unobserved ones, where |Ki| is size of vocabulary of i. Under the new setting, we need to differentiate words in different types of units by giving them different values. We thus obtain the prediction formula by using the weighted combination of words occurring in the two types of syntactic units:

1 nlp.stanford.edu/software/nndep.shtml

where Si is the set of sentiment units and Ti is the set of topic units in post i, s is any sentiment unit in Si and t is any topic unit in Ti, Vk is the feature vector of component word k, and 1 |Si| and 1 |Ti| are the weights of words in sentiment units and topic units, respectively.

4.3 Integrating Baseline Predictor

Koren [2009] suggested a baseline predictor be incorporated into LFM for better generalization. Different from the term of user-item interaction, the baseline predictor embodies general properties of users and words, which takes the first-order form b = ω+bu + ∑k∈Ki bk, where constant ω represents overall sentiment orientation of all the posts in corpus, variables bu and bk indicate the observed deviations of user u and word k in post i, respectively, from the value of ω. The prediction formula by integrating the baseline predictor becomes:

4.4 Model Training and Inference

Based on formula 6, we rewrite formula 4 to obtain the final objective function as follows:

where Regularizer+ = λ ∑u b2u +λ ∑k b2k +Regularizer. The objective function is convex. We search for its minimum using stochastic gradient descent [Bottou, 2004]. All the variables in the parameter space Θ = (bU, bK,M,V) can be estimated automatically, where bU = {bu} and bK = {bk} are bias sets, M = {Mu} are user latent features, and V = {Vk} are word latent features. The set λΘ = {λ, λU, λK} contains the regularization coefficients, for which λ is tuned using the development set, λU and λK are fixed via δ2 δ2U and δ2 δ2K which are set empirically.

Given a new post i of a user u, its personalized sentiment score ˆxui is predicted by formula 6. Then, we use sigmoid function σ(x) = 1 1+exp(-x) to transform it into a distribution. If σ(ˆxui) ∈ [0, 0.5), we classify the sentiment of the post as negative, or as positive if σ(ˆxui) ∈ [0.5, 1].

5 Experimental Evaluation

5.1 Datasets and Setup

Large scale microblog corpus annotated manually is not available. Since emoticons are personal tags given by writers, they are assumed suitable personalized sentiment labels [Go et al., 2009; Pak and Paroubek, 2010]. We selected 33 frequently used positive emoticons and 57 negative ones to derive ground-truth polarities. We crawled the microblog posts of 281 Sina Weibo users and 674 Twitter users using Weibo API 2 and Twitter API 3 , respectively. We obtained 43,250 Weibo posts and 48,563 tweets, each containing one positive or negative emoticon. For Weibo corpus, we used an effective Chinese tokenizer 4 for word segmentation. For both datasets, Stanford POS tagger and neural network dependency parser were employed for POS tagging and dependency parsing, respectively (see Section 4.1). The Chinese sentimental words ontology bank 5 and NRC's EmoLex and MaxDiff Twitter Sentiment Lexicon 6 were used as sentiment lexicons for Weibo posts and tweets, respectively. The statistics about the two datasets are shown in Table 1.

We used 10-fold cross validation for evaluation, where 8 folds were for training, 1 for development and 1 for test. We implemented our models based on the generic factorization tool SVDFeature 7 . A common issue with microblog data is the imbalanced sentiment class distribution [Li et al., 2011b; Liu et al., 2013]. We re-sampled the training instances for each user to balance the proportion of positive and negative posts while keeping the development and test data are intact. We used geometric mean [Kubat and Matwin, 1997] as the metric for evaluation considering imbalanced nature of our dataset, which was defined as G-mean = √Sensitivity ×Specificity, where Sensitivity is the true positive rate and Specificity is the true negative rate.

5.2 Experiments and Results

Fixed parameters setting

We optimized the f parameter via validation on the development set by performing a grid search on all values of 10 ∗ x with x ∈ {1, 2, ..., 10}. Basically the performance was not sensitive with respect to f, and we fixed f = 60 which is slightly better than other choices. We tuned λ using the development data and fixed it as 1.0e-4. Since λU and λT are calculated by δ2 δ2U and δ2 δ2T (see formula 4), we set the two ratios to fixed values as 1.0e-3 as we found that varying them just influenced the results slightly.

2 open.weibo.com/

3 https://dev.twitter.com/overview/api

4 ictclas.nlpir.org/

5 ir.dlut.edu.cn/EmotionOntologyDownload.aspx

6 www.saifmohammad.com/Lexicons/

7 svdfeature.apexlab.org/wiki/Main Page

Effects of decomposition and following relation

We examined the effect of word-level decomposition for dealing with feature sparsity. We compare the ratio of non-zero features of each user before decomposition, i.e., observed posts # total posts # , and that after it, i.e., observed words # total words # . The ratios are displayed in Figure 3. Overall, the non-zero rate of the personal data raised considerably from 0.36% to 2.9% for Weibo data and from 0.15% to 1.26% for Twitter data. This implies that the decomposition is helpful to alleviate the sparsity.

We also studied how the depth of following relation considered can influence the performance. Here we use the formula C = ∑ni=1 1i Ci to calculate the entry values of connection matrix up to n level of depth in the following relation. The strength of connection is decayed by a factor of 1i for the i-th level relation. Figure 4 shows that our model achieved best result when n=2 on Weibo and n=1 on Twitter. This is because the user relation on our Twitter data is much denser. The result seems to indicate that the first-level connection is sufficient for Twitter, but followers one more step deeper is helpful for Weibo, and more depth may bring too much noise. But overall the impact of depth does not appear very strong.

Comparison of different configurations

We compared the performance of five different settings: (1) Basic: direct application of LFM using the original userpost matrix; (2) BOW: our bag-of-words LFM without considering dependency relation and user following relation, which is equivalent to modifying the reviewer-product-review model [Li et al., 2011a] to remove the product dimension not needed; (3) Follow: our model that considers following connection and uses only bag of words; (3) Depend: our model that considers dependency-based syntactic units but not using following relation; (5) Full: our fully configured model.

As shown in Table 2, Basic performs the worst on G-mean due to sparsity of the user-post matrix. Other models with the decomposition appear much better. This indicates that alleviating matrix sparsity is effective to the personalized sentiment classification because many word-level features can take effect after the decomposition. Since BOW only adopted linear combination of words and neglected the importance of word dependencies and social effects, it performs the worst among other contrastive models. Follow improves BOW by enhancing the individuality of target user by interests of connected users, but it still can not compete with Depend and Full on Gmean due to its coarse sentiment-topic representation. Higher G-mean of Full implies that both dependency and following relations are complementary to improving sentiment model.

Comparison of different approaches

We compared our fully configured model with some traditional and advanced baselines all based on the same setup of our datasets. We first compared to standalone models which can be categorized as non-personalized and personalized:

- The non-personalized models did not distinguish user data. We used (1) bag-of-words SVM [Pang and Lee, 2005] with word frequency features; (2) the non-negative matrix tri-factorization based approach with lexicon prior knowledge [Li et al., 2009] named MFP as conventional and strong non-personalized baselines, respectively.

- We used the classifier-fusion framework for multidomain sentiment classification [Li and Zong, 2008] to simulate personalized sentiment model based on SVM, named as PSVM. The model was obtained by training a meta-classifier with input attributes that are the output of base classifiers. We manually grouped users into eight domains based on the occupation in their profiles (e.g., finance, education, etc.) or their following relation when the occupation is not given, and then trained a base classifier for each group of users.

We also compared to two ensemble-based models: (1) Co-train: Li et al. [2011b] proposed a feature-level ensemble method using co-training. The most confident positive and negative results are selected from the outputs of the two classifiers each learned from a subspace of features; (2) SNM (SVM+Naive Bayes+Maximum Entropy): Kittler et al. [1998] proposed an ensemble method with majority vote. So we combined SVM, Naive Bayes and Maximum Entropy with this voting method where the ensemble is of model level.

We tuned all these models on the development set to obtain their optimal parameters. The results of comparison on test set are displayed in Table 3. Compared with standalone methods, ours is obviously higher than both non-personalized and personalized baselines in terms of G-mean. It improves 25.2% over SVM, 18.0% over MFP and 10.2% over PSVM on Weibo dataset, and more strikingly improves 35.7% over SVM, 32.7% over MFP and 27.9% over PSVM on Twitter dataset. This indicates the effectiveness of our personalized sentiment model. PSVM outperforms SVM by 13.6% and 6.1% on Weibo and Twitter data, respectively, which manifests that the personalized data based on different user groups can better distinguish users' personal sentiment. Also, we find ensemble models Co-train and SNM outperform standalone methods, but still clearly worse than ours especially on Twitter dataset. These again verify the advantages of our personalized sentiment classification model.

Significance test

We conducted two-tailed pairwised t-test on G-mean between our model and approaches in Table 3 and among the different configurations in Table 2 by running 10 times 10-fold crossvalidation based on the re-partitioned folds each time [Han, 2005]. Therefore, we obtained 100 sample scores per model. As shown in both tables, most pairs of results examined are significantly different from each other. This confirms the effectiveness of our approach. It seems that our model using follow relation could be further improved as it cannot significantly outperform BOW. It might be because the user connectivity on our Weibo dataset is too sparse, but that on our Twitter dataset is much dense. We leave this for future study.

6 Conclusion

In this work, we focus on the personalized sentiment classification for microblog users. We proposed novel extensible latent factor models to automatically capture the latent individuality from the underlying context of sentimental microblog posts. In particular, our approach decomposed microblog posts into word level for the sparsity of personal data, and considered dependency relations between words for better estimating the word features; it also integrated user's following relations to strengthen the catch of individuality for target users. Experimental results on two real-world microblog datasets show that the performance of our method is very promising and significantly outperforms state-of-the-art baseline algorithms.

Acknowledgments

This work was supported by the State Key Development Program for Basic Research of China (2011CB302200-G), National Natural Science Foundation of China (61370074, 61402091), Fundamental Research Funds for the Central Universities of China (N130604002), General Research Fund of Hong Kong (417112), Shenzhen Fundamental Research Program (JCYJ20130401172046450, JCYJ20120613152557576) and RGC Direct Grant (417613).