Recommendation vs Sentiment Analysis: A Text-Driven Latent Factor Model for Rating Prediction with Cold-Start Awareness

1 Introduction

Review rating prediction is a fundamental problem in the field of sentiment analysis and opinion mining. Detecting users' sentiment polarity or intensity (e.g., 1-5 stars in Yelp and 1-10 stars in IMDB) about all kinds of products from vast amount of reviews has recently drawn close attention from research communities due to its importance to wide range of applications, such as product recommendation, product quality tracking and public opinion mining.

The problem has been approached from two points of views in the literature, i.e., Recommender Systems (RS) and Sentiment Analysis (SA). The RS-based approach typically adopts matrix factorization techniques, which is also known as collaborative filtering, by modeling the inner product of user and product factors [Salakhutdinov and Mnih, 2007]. In this approach, most of its variants focus on capturing user-product interactions while ignoring the valuable content of reviews [Koren, 2008] or just use it as auxiliary information to enhance the representations of users and products [Zhang et al., 2014b]. The SA-based approach has largely regarded the problem as a multi-class classification task focusing on text content following Pang et al. [2005]. Under this direction, most studies rely on handcrafted features and lexicons for effective learning performance, which however is biased and labor intensive. More recently, people using Deep Neural Networks (DNNs) to learn discriminative representations from text has realized the importance of incorporating user/product information [Tang et al., 2015a; 2015b]. However, DNNs typically have much more parameters to estimate, which lead to high model complexity, exhaustive parameter tuning and heavy reliance on word embeddings. Besides, DNNs easily over-fit limited training data and cannot be generalized well when the user-product interactions are sparse in common case.

∗Work done when this author was affiliated with Qatar Computing Research Institute

† kfwong@se.cuhk.edu.hk {MoE Key Laboratory on High Confidence Software Technologies, China}, chengqi.zhang@uts.edu.au

Compared with DNNs, RS-based methods such as Latent Factor Models (LFM) are simple, easy to train and has achieved promising results on review rating prediction [Jin et al., 2016; Zhang et al., 2014b]. However, built from the useritem rating matrix, they encounter challenges when review content has to be considered. For example, different users providing similar reviews on a movie might rate it differently, or they might give it the same rating while writing very different reviews, depending on how strict/lenient they are, or how they like to convey their opinions. This presents the influence of user-specific preference on rating. Similarly, different products given similar review opinions might receive different ratings or receive the same rating but with different reviews, depending on the specific properties of the products. Therefore, it is difficult to knowwhat factors beyond the global user-product interactions really matter to the rating.

Another challenge is the existence of "cold-start" users and products commonly encountered in social media environment, which can decrease the prediction performance dramatically due to the lack of enough reviews and ratings for training [Zhang et al., 2014a; Xu et al., 2015]. Existing works [Li et al., 2011; Tang et al., 2015a; 2015b] cannot learn the representations well for inactive users and unpopular products, and fail to deal with newly entered users or products that do not contribute any review.

In this paper, we propose a simple, extensible Text-driven Latent Factor Model (TLFM) for review rating prediction. We model user- and product-specific LFM components separately and then jointly optimize the two components in a unified framework. Thereafter, TLFMwill be further optimized by a novel Pairwise Rating Comparison (PRC) strategy to alleviate the cold-start problem in product reviews, which was not considered in SA-based methods. The intuitive idea is to enrich the rating information by using the difference between the expected result and ground truth in a rating comparison as supplementary information to calibrate parameter estimation. The main contributions of our paper are three-fold:

• We present a simple, extensible review rating prediction model to capture semantics of review text, user preferences and product characteristics based on a variant of latent factor model.
• We propose a novel cold-start optimization strategy to calibrate the parameter estimation for cold-start users and products which suffer from insufficient ratings. In addition, two mechanisms, AvgUI and UnkUI, are presented for the unseen users and products.
• Our comparative results on three public datasets show that TLFM outperforms state-of-the-art methods especially SA-based approach for review rating prediction.

2 RelatedWork

Sentiment analysis on subjective documents such as tweets and product reviews has been widely studied [Pang and Lee, 2005; Feng et al., 2011; Jiang et al., 2011]. Most of these work ignored the crucial characteristics of users and products which make significant influences on sentiments. Tang et al. [2015b] presented a neural network model that took user information into account, and later they further considered product information and developed a User Product Neural Network model (UPNN) which achieved state-of-the-art performance on rating prediction. However, the number of parameters for each user/product preference matrix in their method was large, which made representation learning inefficient and hard to tune. Besides, learning was ineffective when the user-product interactions were sparse.

LFM, a kind of collaborative filtering method, has drawn great attention in recent years for its promising recommendation performance in Netflix Prize competition [Koren, 2008; Bell and Koren, 2007]. Compared to Neural Networks, LFM is easy to extend and learn, which also has many variants, such as comments rank [Agarwal et al., 2011] and tweet recommendation [Song et al., 2014]. Recent advances are shifting towards utilizing collaborative filtering methods to make sentiment prediction. Song et al. [2015] and Wu et al. [2016] proposed personalized sentiment classification models by considering both microblog users and their social relations. Li et al. [2011] developed a user-product-word tensor factorization model for review rating prediction. However, these studies did not consider cold-start problem, which is very common on review sites and usually influences system performance. Although some LFM methods have considered text, they mostly use the content of reviews as auxiliary information, such as enhancing the interpretation of LFM by extracting user and product features from reviews [Zhang et al., 2014b] or using DNN-based text representation to guide factors estimation [Kim et al., 2016].

Cold start is a challenging issue. Zhang et al. [2014a] proposed a context-aware semi-supervised co-training algorithm for tackling cold start in product recommendation. However, context (e.g., user occupation and movie genre) is not always available and the resulting models are hard to generalize. Xu et al. [2015] utilized active users/popular products with high occurrence frequency to enhance the representation of coldstart users/products. But it is tricky to assign an appropriate threshold to identify cold-start users/products. In this work, we propose a pairwise rating comparison optimization strategy without need to identify cold-start users/products, based on our intuition that the ratings of all users/products are useful to provide clues in comparison. Meanwhile, the factors for all the users and products can be calibrated, which can further improve the performance.

3 Preliminaries

In this section, we first introduce some frequently used notations, and then provide an overview of the Latent Factor Model, which paves way for proposing our TLFM method.

3.1 Notations

For a typical online review website such as Yelp or IMDB, we would have a set of users U writing reviews on a set of products I. We use yui to denote a rating user u ∈ U gives on a product i ∈ I, which is typically associated with a textual review rui on i written by u. The ratings can be integers ranging from 1 (star) indicating no interest to 5 or 10 (star) indicating a strong interest. Given the training set T = {yui|rui, u, i} where yui is known, we want to learn a function fΘ which predicts the most likely rating ̂ yui for each review in the test set T ′ = {yui|rui, u, i} where yui is unknown. We can formulate our review rating prediction task as below:

where function fΘ indicates a predictor with parameters Θ that can be learned by minimizing the differences |yui - ̂ yui| of the estimated rating ̂ yui and the real rating yui over T .

3.2 Latent Factor Model (LFM)

Latent factor model (LFM) is a kind of model-based collaborative filtering method, which is designed for improving the prediction accuracy of the Netflix movie recommendation systems [Bell and Koren, 2007; Koren, 2008]. LFM is trained only based on observed ratings, which can be defined as:

where the observed rating is factorized into four components: global average rating µ, user bias bu, product bias bi and user-product interaction qT i pu that captures user u's personalized preference on product i. The user-factors vector pu ∈ RK and product-factors vector qi ∈ RK usually have a low-dimensional representation in the same factor space and K ≪ min{|U|, |I|}. The LFM only considers user and product information without considering text content which actually dominates sentiment expression in subjective text.

4 Text-driven LFM for ReviewRating

Existing RS-based approaches using LFM focus on modeling user-product interactions, which is not text-driven even though text content might be considered as an auxiliary resource in some of the variants [Zhang et al., 2014b; Kim et al., 2016]. In this section, we propose text-driven LFMs which learn the semantic factors of words directly from the review text. We first present a baseline model, a user-specific model (ULFM) and a product-specific model (PLFM), and then compose them into a fully-configured text-driven model.

4.1 Baseline Model

The baseline model computes rate scoring from general perspective without considering the specific influences from different users and products on review texts. We first present a baseline model which embodies general properties of words and takes the first-order form as follow:

where µ is global average rating, W(rui) is the set of words in review rui, bw is the word bias and αw = 1 |W(rui)| is the normalization term. Therefore, sentiment intensity can be calculated by accumulating the effects of word biases.

4.2 User Latent Factor Model (ULFM)

Review text often reflects user's specific individuality due to their frequently embedded language habit, personal character, opinion bias and so on. Therefore, we propose a Userbased Latent Factor Model (ULFM) which can be represented as a linear combination of a baseline component and a userspecific component as follow:

where the baseline component captures common sentiment knowledge and the user-specific component explicitly captures user influences. We mainly consider user-sentiment consistency and user-text consistency which were first discussed in DNN-based sentiment model [Tang et al., 2015a].

User-sentiment consistency. A user has rating preferences which are independent of the rated products. For example, a critical user tends to give lower ratings, but a lenient user favors giving higher ratings, which is captured by user bias bu. If global average rating µ = 3, the user with bu = +1 is more lenient than the user with bu = -1, since 3+1 > 3-1.

User-text consistency. A user has sentiment specific word preferences. For example, a critical user will use the word "good" to express a much more satisfying attitude than a lenient user, which can be captured by the user-word interaction. Specifically, we decompose review rui into word semantic level by representing each word w ∈ W(rui) as a K-dimensional factor vector vw. Then, user-word interaction vTw pu can capture user u's preference on word w. Finally, user-text consistency can be captured by averaging all userword interactions vTw pu (w ∈ W(rui)).

4.3 Product Latent Factor Model (PLFM)

Based on the model symmetry, we can easily derive a Product Latent Factor Model (PLFM) by explicitly considering product-specific information and common sentiment knowledge. This PLFM focuses on capturing sentiment expression on specific target (i.e., event or product), which is actually the task of target-dependent sentiment analysis [Jiang et al., 2011; Vo and Zhang, 2015]. Our PLFM can be formulated as a linear combination of a baseline component and a productspecific component as below:

where the product-specific component explicitly captures product-sentiment consistency and product-text consistency:

Product-sentiment consistency. A product has rating preferences determined by its characteristics. That is, a highquality product tends to receive high ratings, but a low-quality one might receive low ratings, which is captured by bias bi. For example, if global average rating µ = 3, the product with bias bi = +2 has a better quality than the product with bias bi = -2, since 3 + 2 > 3 - 2.

Product-text consistency. A product has specific preferences on word selection. For example, people use "comedy" and "family" to evaluate the movie of "Mr. Bean's Holiday", but use "action", "fantasy" and "adventure" for "The Hobbit". This product-text consistency can be captured by averaging all product-word interactions vTw qi (w ∈ W(rui)).

Tang et al. [2015a] examined three public benchmark datasets (i.e., IMDB, Yelp 2014 and Yelp 2013) and verified the general existence of these consistencies.

4.4 The Unified Text-driven LFM (TLFM)

Here, we combine ULFM and PLFM smoothly under a unified RS-based framework, which can be formulated as below:

where base = µ + bu + bi + ∑w∈W(rui) αwbw and function g(x) = 1+ C-1 1+e-x makes outputs within the range of valid rating values [1, C] by reformulating the sigmoid function, say, if x → +∞ (or x → -∞), g(x) = C (or 1).

Finally, we obtain our objective function that is to minimize the sum of squared errors with a regularization term:

where the first term strives to fit the given ratings, and the last term avoids over-fitting by penalizing the magnitudes of the parameters Θ = {{pu}, {qi}, {vw}, {bu}, {bi}, {bw}} in the model, weight λ controls the strength of regularization. With the estimated Θ = argminΘ L, we can predict the rating for each testing instance in the test set T ′ using formula 6.

5 Optimization for Cold Start

In this section, we first clarify cold-start problem in review rating task, then present a Pairwise Rating Comparison strategy (PRC) for alleviating the problem, and finally optimize TLFM with the PRC strategy.

5.1 Cold-Start ProblemPreliminaries

To improve the prediction quality for the unseen (or inactive) users and unseen (or rarely rated) products is challenging. The lack of ratings from these users/products may weaken the parameter estimation with respect to the latent factors. Technically, this is referred to cold-start problem [Zhang et al., 2014a; Xu et al., 2015].

For dealing with unseen users or unseen products, inspired by [Tang et al., 2015a] we adopt two solutions called AvgUI and UnkUI. The AvgUI averages over all the observed pu and bu or qi and bi as the representations of new users/products. The UnkUI method learns a shared "unknown" representation for new users/products by randomly drawing ξ reviews as their alternative training instances.

For inactive users and rarely rated products, exiting works [Zhang et al., 2014a; Lin et al., 2013] resort to context information such as attributes (e.g., movie genre or user occupation), implicit feedbacks (e.g., clicks), social relation, etc. Such kind of context is not always available and the resulting models are hard to generalize. With the data at hand, we attempt to improve parameter estimation via a Pairwise Rating Comparison strategy (PRC) which is detailed below.

5.2 Pairwise Rating Comparison Strategy (PRC)

The basic idea of our strategy is to minimize the difference between actual difference and expected difference via rating comparisons to calibrate the factors of users/products. Since there exists no definite boundary between cold-start users/products and active users/popular products, we try to calibrate factors for all the users and products because coldstart users/products can also contribute ratings. Specifically, let rui and rvi be any two reviews given by different users (i.e., u and v) on the same product i, we can build a pair of rating comparisons: actual difference yui - yvi and expected difference ̂ yui - yvi. The intuitive idea is to make expected difference approximate actual difference as close as possible over the whole training set T . We ignore regularization term temporarily to sharpen the focus and obtain the objective function which balances generalization and specificity by combining the two terms in a linear way as below:

where |Ti| is the number of reviews received by the product i and NI is a normalization term. As we can see, this objective function can be viewed as a weighted version of formula 7.

Similarly, we can also define actual difference yui - yuj and expected difference ̂ yui - yuj with the same user u and different products, and obtain another objective function:

where |Tu| is the number of reviews written by the user u and NU is a normalization term. Intuitively, |Tu| and |Ti| reflect the popularity of user u and product i, respectively [Li et al., 2011]. In this work, we denote |Tu| NU and |Ti| NI as the relative importance of training instance from user and product perspectives, respectively, and define the constant NU = 1 |U| ∑u∈U |Tu| and constant NI = 1 |I| ∑i∈I |Ti|.

5.3 PRCOptimization for TLFM

Based on a smooth linear combination of formulas 8 and 9, we rewrite formula 7 to obtain the final objective function:

where Cui = σ( |Tu| NU ) + σ( |Ti| NI ) can be considered as the weight of the observed rating yui and σ(x) = 1 1+e-x is sigmoid function for normalizing relative importance into (0.5, 1), so we can easily derive Cui ∈ (1, 2). Parameters Θ can be learned by stochastic gradient descent [Bottou, 2003], which searches for minimum by updating the related parameters in the negative direction of the gradient as below:

where ˆ eui = Cui(yui - ˆ yui) is the weighted difference between real rating and prediction, and ˆ g = (̂ yui-1)(C-̂ yui) C-1 is derived from g(x)′ . Intuitively, the inputs with popular users and products will have higher weights (i.e., higher Cui) since popular users and products usually have better estimation on their factors. Our model will learn more confidently from the important observations with higher weights, which can lead to better performance and alleviate the cold-start issue.

6 Experiments and Results

6.1 Settings

We experiment on three public datasets 1 : IMDB, Yelp 2013 and Yelp 2014. The datasets are tokenized by Stanford CoreNLP [Manning et al., 2014] and split into training, development and testing sets with exactly the same 80/10/10 split as [Tang et al., 2015a]. The statistics is presented in Table 1.

We evaluate the performance of predictions by Root Mean

Squared Error: RMSE = √∑yui∈T ′(yui - ̂ yui)2 /|T ′ |.

1 http://ir.hit.edu.cn/˜dytang/paper/acl2015/dataset.7z

We optimize the factor number K via validation on the development set by performing a grid search on all values of 10 ∗ x with x ∈ {1, ..., 10} and display results in Figure 1. We see that the performance becomes stable when x = 10, so we set K = 100. We set λ = 1e - 4, and the number of iterations as 50 according to our observation on the best RMSE scores on the development set. All the parameters are initialized randomly from a uniform distribution on the interval (0, 0.1). Our experiments are conducted on a commodity Windows 7 PC with Core i7-6700 CPU and 16GB RAM.

6.2 Comparison of different methods

We compare TLFM with some traditional and advanced baselines as shown in Table 2, and separate results into three groups: (1) text-independent methods; (2) text-based methods; (3) the methods using text, user and product information.

1. Majority considers the rating value which is the majority in training set as the predicted rating of each review in test set; LFM is the matrix factorization method trained on user-product rating matrix (see formula 2).
2. Ngram is a support vector machine (SVM) classifier trained on unigram, bigram and trigram features [Fan et al., 2008]; TextFeature is the feature engineering method using extracted text features including word n-grams, sentiment lexicon features, and negation features for training a SVM classifier [Kalchbrenner et al., 2014]; AvgWordvec is obtained by training a SVM classifier with input features that are averaged word embeddings in a review [Mikolov et al., 2013]; SSWE averages Sentiment-Specific Word Embeddings as review text representation, and then trains a SVM classifier; Paragraph Vector is a SVM classifier trained on paragraph representations of documents [Le and Mikolov, 2014]; RNTN+Recurrent learns sentence representations with Recursive Neural Tensor Network [Socher et al., 2013] which are then fed into a recurrent neural network, where the hidden vectors are averaged for classification.

(3) JMARS is a probabilistic model based on collaborative filtering and topic modeling, which considers user and aspects of a review [Diao et al., 2014]; TFM [Li et al., 2011] is a linear model that combines the entries estimated from a user-product-word tensor factorization model; UPNN is a neural network model which modifies word embeddings in the input layer with user/product preference matrix, and then concatenates user/product vector with generated review representation via softmax layer [Tang et al., 2015a]; TLFM is our proposed text-driven LFM, and TLFM+PRC additionally considers cold start with PRC.

In Group 1, LFM outperforms Majority by using user and product interactions, but it still cannot compete with textbased methods. In Group 2, SVM classifiers with Ngrams or handcrafted text features outperform AvgWordvec, SSWE and Paragraph Vector in most cases that are trained on word embeddings. RNTN+Reccurent performs the best by modeling document with semantic composition. InGroup 3, UPNN and TLFM outperform JMARS and TFM by explicitly modeling user preferences and product characteristics, which implies considering the four consistencies is effective. Our TLFM is better than UPNN since the latter needs to learn much more parameters regarding preference representations in different network layers, which makes it difficult for parameter tuning and achieving comparable performance on our sparse rating matrices. Also, UPNN heavily relies on word embeddings which may be not well trained on the limited datasets. This is highly encouraging, indicating the simplicity and effectiveness of our method. Besides, PRC strategy achieves further improvements, which suggests that cold-start problem can be alleviated.

6.3 Effect of the four consistencies

We study the effects of consistency consideration on rating prediction, including user-sentiment consistency (us), usertext consistency (ut), product-sentiment consistency (ps) and product-text consistency (pt). We compare different configurations of UPNN and our TLFM+PRC in Table 3.

It is clear that the partial configurations cannot compete with full models indicating all the consistencies need to be considered. In our method, we find that ut+pt>us+ps, implying that ut and pt are more important than us and ps, because text provides better sentiment clues. In addition, us+ut>ps+pt indicates that user preferences are more impactful than product traits. TLFM outperforms UPNN under most configurations, which verifies its effectiveness on capturing the effects of users and products.

6.4 Effect of unseen users and products

We also present the prediction results of our methods (i.e., UnkUI and AvgUI) for the unseen users or products. We first randomly select 10% users and products from test set, replace their names with unseen names so as to obtain a coldstart test set. We set ξ as 200 for UnkUI. As shown in Table 4, both AvgUI and UnkUI perform much better than UPNN (see Table 2), which verifies the effectiveness of our methods.

6.5 Effect of user and product popularity

We further study the influence of user/product popularity. User or product popularity is determined by the number of times a user rates or a product is rated. We partition users/products into four equal-sized bins according to their popularity in training set as shown in Figure 2 where the popularity from low to high is given from bin 1 to 4. We then group the test instances corresponding to these bins, and obtain the performance in Table 5. We find that the bins with more popular (active) users/products usually have better results since user/product factors are estimated more accurately. TLFM+PRC improves TLFM substantially over all the bins, which indicates the effectiveness of our PRC strategy on alleviating cold-start problem.

6.6 Complexity analysis and running time

Let |T | be the size of training set, L be the average number of words in each training instance, K be the factor number and N be the number of iterations. The time complexity of TLFM+PRC mainly lies in updating parameters Θ. Its overall time complexity is O(N|T |(K + 1)(L + 2)). UPNN is based on a Convolutional Neural Network with three filters (see [Tang et al., 2015a]). Its complexity mainly lies in producing the convolutional features. Let dU be the output length of the multiplicative composition between user preference matrix and word embedding, and dP be the output length of the multiplicative composition between product preference matrix and word embedding, X be the dimension of word embeddings and S be the output length of each filter. In this work, we have X = 2K, so the overall time complexity of

���� �������� ��������

(a) Grouped users

(b) Grouped products

Table 5: Cold-start performance of users and products.

UPNN is O(N|T |(X+6S)(dU +dP )L). Obviously, UPNN has much higher time complexity, which results from much more parameters regrading preference representations.

Meanwhile, we also compare the running time between TLFM+PRC and UPNN on the training sets. We run the code of UPNN with suggested settings in [Tang et al., 2015a] on the same experimental environment as TLFM+PRC. We then have the results: UPNN (IMDB: 441.84 min, Yelp14: 450.64 min, Yelp13: 158.65 min) and TLFM+PRC (IMDB: 7.21 min, Yelp14: 9.67 min, Yelp13: 3.12 min). We can find that UPNN is much more time-consuming than TLFM+PRC for training, which indicates the high efficiency of our method.

7 Conclusion

We proposed a simple, extensible text-driven latent factor model for review rating prediction, which is focused on capturing interactions between review content and user preferences/product characteristics. We also addressed two cases of "cold-start" by developing AvgUI and UnkUI for the unseen users/products and presenting a PRC strategy to calibrate the factors for the inactive users and rarely rated products. The experimental results on IMDB and Yelp datasets show that our method outperforms state-of-the-art baselines.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 61370074, 61402091) and partially supported by the UGC, HK, under the GRF initiative (#14232816) and ITF, HK, (#ITP00416PL).