Your Tweets Reveal What You Like: Introducing Cross-media Content Information into Multi-domain Recommendation

1 Introduction and RelatedWork

In the era of big data, recommender system plays an essential role in improving user experiences on the Internet, particularly in E-Commerce. A well-performing recommender system enables users to find their favorite products. Therefore, it is necessary to model how a user evaluates the products, namely rating prediction.

CF algorithms are widely used to tackle rating prediction problem with users' rating history in recommender systems. As a vital step in most CF algorithms to get the latent embeddings of users and items, MF achieves remarkable performance [Rennie and Srebro, 2005]. Later, discrete factorization is used to mend MF [Porteous et al., 2008]. Ruslan and Andriy propose Probabilistic Matrix Factorization (PMF), where a probabilistic model is introduced [Mnih and Salakhutdinov, 2008]. Alex et al. describe a unified Bayesian approach, named as Collaborative Bayesian filtering (CoBaFi) [Beutel et al., 2014]. However, these algorithms are challenged in the data sparsity problem when users or items rarely have rating history.

∗Contact author

In another branch of research, content-based algorithms can overcome the obstacle of limited historical data, because they are driven by the attribute features of items and users, which are able to recommend items with little users' history information. According to this idea, several studies introduce content information, such as user reviews on products, into CF algorithms and achieve stronger results. Topic models, including Latent Dirichlet Allocation [Blei et al., 2003], are applied to discover the aspects of different products from reviews. McAuley and Leskovec propose an algorithm named Hidden Factors and Hidden Topics (HFT) that unveils topics correlated with the 'hidden factors' of products and users [McAuley and Leskovec, 2013]. Besides, review sentiment is used in Diao et al.'s work [Diao et al., 2014]. Reviews are helpful in tackling the problems in the recommendation.

Although additional information from recommender system enhances the prediction results of users with limited rating history, it cannot cope with the cold-start problem (new users without any rating history information). Hence some studies try to utilize cross-domain information (e.g. user's rating history in other domains) in transfer learning algorithms [Elkahky et al., 2015; Zhang et al., 2016b; Man et al., 2017]. Some other studies try to introduce crossmedia information, such as user relationships, user's metadata and preferences in social media, into rating prediction. We focus on employing user's content information in social media in this study. Here are some related studies: based on the idea that users may have similar interests with their friends, Yang et.al. propose Truster Matrix Factorization (TrustMF) [Yang et al., 2013], where the integration of user relationships reinforces rating prediction. The compactness degree of user relationships is taken into consideration by [Wang et al., 2017b]. Meanwhile, user's preference in social media is mined for the recommendation. User's 'like' history and other behavior are introduced into rating prediction algorithms [Yan et al., 2015]. User metadata, such as user profile, is applied in [Deng et al., 2013]. Recent deep learning solutions for cross-media information enhanced recommendation and achieve good performances [Zhang et al., 2016a; Wang et al., 2017a; He and Chua, 2017]. Encouraging results demonstrate that the cross data does facilitate user profiling and rating prediction. Not only new users, but also users with rich historical data in recommender system benefit from the introduction of the additional information.

It should be noticed that the cross-media features applied in previous studies are on-topic information, which directly related to users' preferences in the target domain or characteristics of target items (e.g. user meta-data, ties, and behavior). Former studies have neglected the off-topic content information in social media since most of it is unrelated to target items or the target domain. For instance, most user tweets are offtopic and ignored in user preference mining. However, we consider that the tweets generated by users in social media are as valuable as other on-topic features, because they assist to learn the embedding of users for profiling. Therefore, user tweets are applied to rating prediction in this study. Our purpose is to confirm whether the omitted off-topic information is effective on improving the rating prediction results.

To shed light on this problem, we propose a framework to capture features from user's off-topic content and include them in MF based algorithms. Within this framework, three feature extraction methods and four MF based algorithms are adopted. The experiments conducted on all yield achieve encouraging results, with further analysis enclosed. The main contributions are listed below:

• To the best of our knowledge, this is the first work that applies user tweets to rating prediction algorithms, most of the tweets are off-topic content information.
• We design a framework to introduce the content information from social media into MF based algorithms, in which user's content features can be directly used to boost recommendation in multiple domains.
• The experimental results in multi-domain show that the off-topic content contributes to significant improvements on various state-of-the-art rating prediction algorithms with different embedding approaches. The proposed algorithms are practically advantageous when dealing with the cold start problem.

The remainder of this paper is organized as follows: A detailed description of our framework is stated in Section 2. Subsequently, in Section 3, we discuss experimental settings and the comparative results. Further analyses are shown in Section 4. The conclusions and the outline of future work are drawn in Section 5.

2 Social Media Content Enriched Framework 2.1 Framework

The task in this paper is rating prediction: to predict a user's rating (an integer ranges from 1 to 5) on a product. The notations are summarized in Table 1. We aim to include user's content information in social media into rating prediction. It then becomes a cross-media problem in two parts: 1) how to introduce the content-based features into rating prediction algorithms; 2) how to extract useful features from social media content. As shown in Figure 1, we propose a twosection framework accordingly: incorporated matrix factorization and feature extraction of content information.

For the former part of the problem, most rating prediction algorithms are based on MF, consisting of user latent matrix U and item latent matrix I. Inspired by previous studies, the user features ⃗ Du learnt from social media content is introduced into user's latent vector ⃗ Uu for MF training (the top part of Figure 1). The modified ⃗ Uu consists of two parts: user embeddings from the content information ⃗ Du and domainspecific user features ⃗ Bu. D marked with grey means it is extracted from social media content. As D will affect the learning of B and I, the training results of them are marked with grey shadows to denote that they are affected by the crossmedia information.

For the latter part of the problem, the length of ⃗ Uu in rating prediction algorithms ranges from 5 to 10 in most conditions. Thus, ⃗ Bu is supposed to be in low-dimension. The map function F is used for user embedding learning, which can be implemented with various feature extraction methods. In our experiment, we examine multiple algorithms in this step.

Note that our main purpose is to investigate if the off-topic content information from social media can support stronger performance of preceding algorithms in the rating prediction. We work with several embedding learning methods and apply them to different MF based algorithms. The two steps are elaborated in next subsections.

2.2 IncorporatedMatrix Factorization

MF is widely used in rating prediction algorithms, in which rating matrix Rm∗n is factorized as user matrix Um∗k and item matrix Ik∗n. The predicted rating R′u,i that user u may give to item i is calculated by Equation 1. b is a bias factor. The objective function LH is Equation 2, which aims to minimize the loss between the real rating Ru,i and R′u,i . || · ||2F denotes the Frobenius norm for matrix and λ is the regularization parameter.

As shown in Figure 1, user feature D leant from their content in social media, is introduced into MF based algorithms. We choose vector concatenation to fuse the embeddings here. The process to introduce D into U is as following:

• Initializing U. ⃗ Bu will be initialized with random normalization offsets before training and ⃗ Uu = [ ⃗ Du, ⃗ Bu].
• Using MF to train U and I. ⃗ Du will be kept while ⃗ Bu is updated in training to get the domain specific features.

The new prediction result R′′u,i is calculated by Equation 3, and the objective function is Equation 4. ⃗ Ii is split into left part ⃗ Iil and and right part ⃗ Iir in Equation 3.

We do not replace ⃗ Uu with ⃗ Du directly, because user's actions in recommender system can assist profiling too. Note that ⃗ Du is held in updating U, so the content features are maintained and they will affect in training I and B. Meanwhile, ⃗ Bu captures a user u's features from his/her rating history. Consequently, the modified MF algorithm can profile a user's content features and rating features at the same time. If the off-topic content information contributes, results of rating prediction will get improved.

We name the as Social Media Content Enriched Matrix Factorization (MF S). Moreover, MF is common in various rating prediction algorithms, so it is convenient to apply the content information to other algorithms similarly. We will elaborate it in Section 3.2.

2.3 Feature Extraction of Content Information

Several embedding learning methods are adopted as F, including Stacked Denoising Autoencoder (SDAE) [Vincent et al., 2010], word2vector (W2V), and paragraph2vector (P2V).

The first method employs SDAE and bag-of-words (BOW), and it goes through two steps to learn ⃗ Du: 1) Learn high-dimension embedding. The words in user content reveal his/her habits and contribute to profiling, so BOW is applied to learn a basic embedding. All of a user u's tweets without stop words and signals are used in training. 2) Compress high-dimension representation to low-dimension. SDAE stands for its powerful ability to compress features, which consists of several Denoising Autoencoder (DAE). As shown in Equation 5 to 7, DAE is an unsupervised neural network which aims to minimize the MSE loss between the input data ⃗x and the predicted ⃗x′ , and ⃗y is the output. In this study, the input of SDAE is BOW embeddings and the output is ⃗ Du:

The second method is based on W2V, a famous word embedding algorithm [Mikolov et al., 2013]. It is handy to control the length of word embedding in training, so W2V is adopted to learn a low-dimension latent vector of each word. ⃗D is calculated by averaging the words in users' posts.

However, W2V only considers the embedding of each word rather than a sentence, the order of words is missed. Le et al. propose a P2V model that jointly learn word and document embeddings [Le and Mikolov, 2014]. So we take one's tweets as a document and use P2V to learn ⃗ Du from it. Similar toW2V, it is easy to adjust the embedding length.

The three methods are employed to get the user embedding matrix D. In general condition, the other part of user embedding, matrix B, is trained by MF introduced in Section 2.2. However, when coping with new users, due to the lack of rating history, MF cannot train the domain specific user matrix B. In this scenario, we have ⃗ Uu = [ ⃗ Du, ⃗ Bu] and ⃗ Bu = ⃗0. Several methods are designed to complete matrix B:

• M1: filling ⃗ Bu with 0, ⃗ Bu = ⃗0;
• M2, M3, and M4: K-Nearest Neighbor based methods is applied. ⃗ Bu is set as ⃗ Bn if the similarity of ⃗ Du and ⃗ Dn is the highest (n ∈ training set). Eulerian distance, cosine similarity, and Manhattan distance are used to calculate the similarity and named as M2, M3, and M4.
• M5: ⃗ Bu is set as the averaged embedding of users in training set.

The cold start problem is a general obstacle in real scenarios. Facilitated by the additional information and user embedding learning methods, the modified algorithms can deal with new users.

2.4 Model Training

Our framework goes through two steps and there are two types of training methods, the loosely coupled and the tightly coupled. The former one conducts the two-step training one by one (to learn user embeddings and then apply them to MF), while the latter sets a global objective function to train the two parts together. Due to the limit of length, we only show the detailed tightly coupled training method of SDAE + MF here, which is named as MF S T.

As shown in Figure 2, SDAE and MF are trained together and the global loss function is Equation 8, which adds SDAE's loss with weight α into MF loss. The rating is predicted one by one and stochastic gradient descent (SGD) algorithm is used in model training. ⃗ Bu and ⃗ Ii are updated by the global loss. Because the neural network is back propagation, half weights are trained by the global loss and others are updated by SDAE loss (marked in Figure 2). ⃗ Du will not be updated directly since it is learnt from SDAE.

Intuitively, tightly coupled method has stronger fitting ability so it is able to get better performance. While we prefer loosely coupled method because it is convenient to reuse the user embedding ⃗ Du trained on user content into multidomains without extra training. So the following experiments will learn ⃗ Du in advance and then conduct MF. Tightly coupled based experiments will be reported in Section 4.2.

3 Experiments and Results

3.1 Datasets

Two platform datasets of overlapped users, Douban 1 and Sina Weibo 2 (a social network like Twitter in China), are examined. These datasets are widely used in cross-media studies [Jiang et al., 2016; Yuan et al., 2013]. Users can rate three types of items in Douban: music, movie, and book. The ratings are integers ranging from 1 to 5 and some are accompanied by reviews. Tweets and user following relationships in Weibo are recorded. There are over 60,000 users in the dataset. Users who have less than 5 tweets in Weibo and 3 ratings in Douban are removed. 21,356 users remained and each user published 1,521 tweets on average. Table 2 lists the statistics of the three domains after filtering.

3.2 Baseline Models and EvaluationMetric

MF, PMF, HFT, and TrustMF are four baseline algorithms. MF algorithm is illustrated in Section 2.2, and the introductions of other methods are as following:

• PMF: incorporates a probabilistic model into MF.

1 www.douban.com

2 www.weibo.com

• HFT: discovers topics that are related to the 'hidden factors' of items and users from reviews.
• TrustMF: makes use of user ties in social media.

We modify above algorithms according to our framework that can cooperate with different MF based algorithms. Except for MF training part, we do not change the objective functions and prediction functions of them. The modified algorithms are named as PMF S, HFT S, and TrustMF S respectively. If the off-topic content in social media is indeed informative, the modified algorithms should get better performance than the originals. Similar to previous studies, we use Root Mean Square Error (RMSE) as the evaluation metric. Higher RMSE indicates worse performance.

3.3 Setting of User Embedding Learning

The content information used here is user tweets in Weibo, most of which are off-topic. As introduced before, k is an essential parameter constraining the length of ⃗ Uu. We conduct some pilot experiments and find the best parameter setting is | ⃗ Uu| = 5 and | ⃗ Du| = 3 in these datasets. The settings of user embedding learning methods are listed below:

1) BOW is calculated based on high-frequency words (top 4,000) as the input of SDAE. The output of SDAE is set as 3 according to | ⃗ Du|. We use four DAE here (the widths are 800, 100, 20, and 3 respectively) and the 4th DAE's output is ⃗ Du. The batch size is 64 and the epoch is 25 in training; 2) The setting of W2V is the same as Mikolov's work [Mikolov et al., 2013]. Words with low frequency are removed (less than 3). To ensure the embedding short enough to be introduced into ⃗ Uu, the length of the word vector is | ⃗ Du|; 3) We apply Ai's PV-DBOWmodel [Ai et al., 2016] in P2V and the length of the embedding vector is the same as W2V.

3.4 Rating Prediction Results

The modified algorithms incorporate users' content features into MF, and reminders are the same as the originals. We tune the parameters (such as λ) to achieve the best performance of algorithms. Prediction experiments are conducted with 5-fold cross-validation. The RMSE results are shown in Table 3.

Basic results show that TrustMF achieves the best performance among the original algorithms on Music domain and Book domain, indicating that user relationships do help in rating prediction. The performance of HFT algorithm is not as good as expected. It applies users' reviews to construct the user and item matrices, while only 48.0% ratings have reviews and the average review length is only 5.76 words. Short and low quality reviews weaken the performance.

Apparently, with any one of the three embedding learning methods, social media content enriched rating prediction algorithms outperform the originals in three domains. Most of the improvements are statistically significant, expect for MF S in Movie domain (The reason is elaborated in Section 4.1). The degrees of improvements are distinct across the algorithms and the most remarkable improvements are ranging from 0.03% to 4.27%. TrustMF S algorithm yields the best performances on Music domain and Book domain, which integrates both social relationships and content features in prediction. PMF S outperforms in Movie domain, with constantly better performances compared to MF S. In these experiments, HFT S achieves the best enhancement.

Comparing the performance of different feature extraction methods, the efficiency of the methods varies in the prediction algorithms. In most conditions, W2V performs worse than P2V. It is due to that the order of words is considered in P2V. SDAE performs the best in most tasks for its twostep fine-grained embedding learning. Meanwhile, whatever embedding approaches are employed, these prediction algorithms achieve obviously better performances.

All of users' tweets are used in these algorithms, most of which are not directly related to the three domains (off-topic), but they are effective in prediction. We believe the reason is that user's tweets reveal his/her traits and hobbies. Similar to users' social ties, the embedding learnt from user tweets facilitate to identify users' preferences by other users with similar interests. Consequently the prediction results are improved. More detailed comparative analyses and case study are shown in Section 4.1 and 4.3.

To summarize, the integration of the social media content assists multiple rating prediction algorithms achieve good performances. The results also indicate that both on-topic and off-topic content (the latter has been omitted in previous studies) are valuable and useful for the recommendation.

4 Analyses and Discussions

4.1 On-topic Tweets Vs. All tweets

Experiments have verified that user tweets are constructive in rating prediction, but we are not sure whether the improvements are solely contributed by a few on-topic tweets or not. A tweet is on-topic if it contains at least a music/movie/book title in datasets (some ambiguous titles are removed, such as Love, Friend). The percentages of on-topic tweets in music, movie, and book domains are 1.88%, 0.38%, and 2.63%. Since there are still some ambiguous titles, part of off-topic tweets are wrongly recognized as on-topic, so the ratios are higher than expected. Additional experiments are conducted to examine the performance when only on-topic tweets are used. SDAE is applied to extract user features for its good performance with the loosely coupled method. The results are listed in Table 4.

From the table, algorithms with all tweets outperform with only on-topic tweets, most of which are statistically significant in Music domain and Book domain. However, the improvements in Movie domain are limited. The sparsity of Movie domain (99.42%) is lower than others (99.86% in Music and 99.88% in Book), which means Movie dataset contains more user logs. A suggested reason is that there is already enough user logs in Movie dataset, so the addition of tweets is fruitless. To verify this assumption, half user-item ratings in Movie dataset are randomly removed (its sparsity is increased to 99.68%) and more experiments are conducted.

As shown in Table 5, the tweets boosted algorithms perform significantly better than the originals in the sparsified Movie dataset. The performances of different algorithms with all tweets are significantly better than with only on-topic tweets too. To conclude, both on-topic & off-topic tweets are effective and off-topic content should not be ignored in rating prediction. These results also indicate that the content information in social media is more helpful in coping with sparse data, likewise in the cold start scenario.

4.2 Tightly Coupled Training Results

The experimental results of applying tightly coupled method to conduct framework training are introduced in this section. Similar to Section 2.4, we modify PMF S and TrustMF S based on this idea too, and the new algorithms are named as PMF S T and TrustMF S T. The best experimental results and are shown in Table 6, which are achieved with distinct values of α ranges from 0.5 to 5 .

As expected, the tightly coupled trained algorithms significantly outperform the loosely coupled in all conditions and beat the original algorithms easily for their stronger fitting ability with the global loss function. The results show that our framework delivers more accurate results with this training method. While these algorithms spend a longer time in training and user embedding ⃗ Dk is domain specific and cannot be applied to other domains. Nonetheless, we can still choose to use the loosely coupled method or the tightly coupled method according to specific scenarios.

4.3 Tweets Analysis: a Case Study

The experimental results show the effectiveness of the offtopic content information in promoting rating prediction. How do the tweets affect the prediction results? We look into the prediction results in Movie domain, and there are two tweets posted by U1 (Uid 1405524) and U2 (Uid 1305835):

U1: I really like funny stories, which always makes me happy... U2: Looking back into what I have seen in recent years, most of them are funny cartoons...

The tweets indicate that U1 and U2 are likely to have a similar preference. It is easy to capture this preference by bag-ofwords or other methods for collaborative filtering. Therefore, the off-topic tweets assist to enhance prediction results. In a word, user profiling based on the content features is performed efficiently in a hidden way for the recommendation.

4.4 Prediction Performance for NewUsers

An obvious advantage of employing cross-media information for rating prediction is that it will be effective on dealing with the cold start problem, especially for new users. The performance of MF S in handling new users is examined here. To get new users, we randomly sample 90% users as training set and the rest 10% as the test set in each dataset. The logs of users in the test set are removed in training set. Most traditional algorithms cannot conduct prediction or recommendation for these users. However, with the help of cross-media information, MF S algorithm can learn user embedding from their tweets. We follow the setting of SDAE in Section 3.3. The methods M1 -M5 to complete matrix B, introduced in Section 2.3, are applied here.

Figure 3 shows the prediction results. M5 performs significantly better than other methods in multiple domains. M1 with the worst RMSE shows that using ⃗0 is a bad idea. The three-similarity based methods perform worse than M5. We attribute it to the sparsity of users: the scale of the training set is not large enough to find a user with same interests to the user u. Comparing the RMSE results of M5 with the results in Table 3, rating prediction results in new users are worse than in normal condition due to the lacking of user logs. Generally speaking, the performance of MF S in new users is not bad. RMSE value is only 0.051 higher than the results in predicting non-cold-start users on average. The incorporation of cross-media information enables the algorithms handle new users, which is essential for recommender systems.

5 Conclusions and FutureWork

In this paper, we propose a novel and effective framework to leverage full content information in social media, including both on-topic and off-topic content, for rating prediction. The framework is easy to understand and implement, meanwhile shows great flexibility with different embedding approaches and different algorithms. It also achieves remarkable results in multiple domains with the modified algorithms. To the best of our knowledge, it is the first work that the off-topic content information is utilized and observed positive effects in the recommendation for rating prediction. In the newuser scenario, equipped with the cross-media content information, greater improvements have been achieved. Based on this study, we suggest that further studies should utilize full content information in cross-platform to the recommendation.

Acknowledgements

This work is supported by Natural Science Foundation of China (Grant No. 61672311, 61532011). And this project is partially funded by the MOE-Microsoft Key Laboratory of Web and Multimedia, Tsinghua University.