Collaborative Topic Regression with Social Regularization for Tag Recommendation

1 Introduction

Tagging systems have been playing very important role for us to better categorize and organize information. For example, Flickr 1 uses tags to label and organize photos, Last.fm 2 adopts tags to categorize artists and music, and CiteULike 3 allows users to tag articles. With the tagging systems, users are able to better organize their own content and find relevant resources (content) more easily.

However, finding the set of proper words (tags) to describe the resources often requires high mental focus. That is why tag recommendation (TR) [Gupta et al., 2010; Wang et al., 2012] has become more and more important on the Internet. With the tag recommendation system, users only need a few clicks to finish the tagging process. Moreover, tags created by various users can be inconsistent and idiosyncratic. Different users might use different words to express the same meaning, which makes it more difficult to utilize the tagging information. Tag recommendation can help to limit vocabulary of tags and thus alleviate the above problems. Furthermore, it can also help to prevent misspelt or meaningless words. Therefore, TR [Wang et al., 2012] has become a very hot research topic in recent years, and many methods have been proposed by researchers.

1 http://www.flickr.com

2 http://www.lastfm.com

3 http://www.citeulike.org

Existing tag recommendation methods can be roughly categorized into three classes [Wang et al., 2012]: content-based methods, co-occurrence based methods, and hybrid methods. Content-based methods [Chen et al., 2008; Lipczak et al., 2009; Shen and Fan, 2010; Lee et al., 2010; Toderici et al., 2010; Chen et al., 2010], directly adopt the content of resources/items, such as abstract of articles, image content and description of images, to perform tag recommendation. Cooccurrence based methods [Benz et al., 2006; Xu et al., 2006; Hotho et al., 2006; Marinho and Schmidt-Thieme, 2007; Sigurbj¨ornsson and van Zwol, 2008; Garg and Weber, 2008; Weinberger et al., 2008; Wu et al., 2009; Rendle and Schmidt-Thieme, 2010] mainly use the co-occurrence of tags among items (i.e., the item-tag matrix) for tagging. Actually, the underlying principle of co-occurrence based methods is similar to that of collaborative filtering (CF) methods [Adomavicius and Tuzhilin, 2005; Zhen et al., 2009; Li and Yeung, 2011]. Because the TR problem is very complex and difficult, neither co-occurrence based methods nor content based methods can achieve satisfactory performance in real TR applications. Hence, the recent trend in TR research is to use hybrid methods [Wu et al., 2009; Sevil et al., 2010; Lops et al., 2011; 2013] which try to combine both item-tag matrix and item content information together for recommendation.

However, it is still a challenge to find an effective way to combine both item-tag matrix and item content information for TR. Furthermore, in some applications there may exist social networks (relations) between items. For example, if we want to tag articles in CiteULike, there are citation relations or other social networks between articles [Li et al., 2011; Wang and Li, 2013]. Typically, two articles with relation between them might be most likely to be about the same topic [Li et al., 2009a; 2009b], and consequently they should have similar tags. Hence, how to effectively integrate social networks between items for tagging is another challenge.

In this paper, we propose some novel methods to solve the above challenges. The main contributions of this paper can be outlined as follows:

• We adapt the collaborative topic regression (CTR) model [Wang and Blei, 2011], which has been successfully applied for article recommendation, to combine both item-tag matrix and item content information for tag recommendation in a principled way.
• By extending CTR, we propose a novel hierarchical Bayesian model, called CTR with social regularization (CTR-SR), to seamlessly integrate the item-tag matrix, item content information, and social networks between items into the same principled model.
• Extensive experiments on real-world data sets show that CTR can outperform the baselines which use only one kind of information, either item-tag matrix or item content information. Furthermore, CTR-SR can effectively utilize the social networks between items to further improve the performance.

2 Problem Statement

Assume we have a set of items W = [w1,w2, · · · ,wJ] to be tagged, where wj ∈ Rd denotes the content (attributes) of item j. For example, if we want to tag articles (papers) in CiteULike, the items are papers, and the content information can be the bag-of-word representation of paper abstract. Assume there are I tags {t1, t2, · · · , tI} which are candidates to be recommended to tag each item. Then we can use a tagitem matrix 4 R = [rij]I×J to represent the tagging information for all the items. rij is a binary variable, where rij = 1 means that the tag ti is associated with itemwj. Otherwise, rij = 0 means that tag ti is not associated with itemwj. The tag recommendation task is to predict the missing values in rj = [r1j, r2j, · · · , rIj]T . Note that we focus on tag recommendation for articles (papers) in this paper. However, our models are flexible enough to be applied in other applications such as image and video tagging because we can also represent the image and video content as bag-of-words.

The content base methods use only the content information for recommendation. For example, if we want to recommend tags for itemwj, we can use the tags from the nearest neighbor in W based on the content similarity. We can also treat each tag as a label and use multi-label methods to train classifiers based on content information.

Co-occurrence based methods use only the item-tag matrix R for recommendation. For example, if ti and tk occur simultaneously in many items' tags and ti is associated with wj, we should also recommend tk to wj. It is easy to see that the underlying principle of co-occurrence based methods is similar to that of collaborative filtering [Adomavicius and Tuzhilin, 2005].

Both content based methods and co-occurrence based methods discard some useful information. Hence, they can not achieve satisfactory performance in real applications.

4 For ease of presentation, we use tag-item matrix and item-tag matrix interchangeably in this paper.

3 Collaborative Topic Regression

Collaborative topic regression (CTR) [Wang and Blei, 2011] combines CF and latent Dirichlet allocation (LDA) [Blei et al., 2003] to perform recommendation. CTR is initially proposed to recommend articles (papers) to users by utilizing both user-article rating information and article content information. In this paper, we adapt CTR to our tag recommendation problem to seamlessly integrate both item-tag matrix information and item content information.

For ease of presentation, we use similar graphical models and notations as those in CTR [Wang and Blei, 2011] for our problem formulation. The graphical model of CTR is illustrated in Figure 1. Assume there are K topics β = β1:K. The generative process of CTR for tag recommendation is listed as follows:

1. Draw tag latent vector for each tag i:

where N(·) denotes the normal distribution, IK is an identity matrix with K rows and columns.

1. For each item j,
2. (a) Draw topic proportions θj ∼ Dirichlet(α).
3. (b) Draw item latent offset ϵj ∼ N(0, λ-1 v IK) and then set the item latent vector to be: vj = ϵj + θj.
4. (c) For each word wjn of item (paper) wj,
5. Draw topic assignment zjn ∼Mult(θj).
6. Drawword wjn ∼Mult(βzjn ).
7. Draw the tagging information rij for each tag-item pair (i, j),

where cij reflects the confidence of rij:

with a and b being tuning parameters and a > b > 0.

We can adopt the maximum a posteriori (MAP) estimation to learn the parameters of CTR. The details can be found in [Wang and Blei, 2011].

It is easy to see that the above process integrates matrix factorization (MF) [Koren et al., 2009] based CF (Equation (1)) for tagging information and topic modeling for item content information into the same principled framework.

4 Collaborative Topic Regression with Social Regularization

By extending CTR, we propose a novel hierarchical Bayesian model, called CTR with social regularization (CTR-SR), to seamlessly integrate the item-tag matrix, item content information, and social networks between items into the same principled model. The graphical model of CTR-SR is shown in Figure 2.

The generative process of CTR-SR is listed as follows:

1. Draw tag latent vector for each tag ti:

1. For each item j,
2. (a) Draw topic proportions θj ∼ Dirichlet(α).
3. (b) For each word wjn of item (paper) wj,
4. Draw topic assignment zjn ∼Mult(θj).
5. Drawword wjn ∼Mult(βzjn ).
6. Draw the social latent matrix S = [s1, s2, · · · , sJ] from a matrix variate normal distribution [Gupta and Nagar, 2000]:

1. Draw the item latent vector for item j from the product of two Gaussians (PoG) [Gales and Airey, 2006]:

1. Draw the tagging information rij for each tag-item pair (i, j),

In the above generative process, S denotes the social latent matrix of size K×J , each column of which is the social latent vector sj for item j, NK,J (0, IK⊗(λlLa)-1 ) in (2) denotes a matrix variate normal distribution [Gupta and Nagar, 2000]:

where the operator ⊗ denotes the Kronecker product of two matrices [Gupta and Nagar, 2000], tr(·) denotes the trace of a matrix, La is the Laplacian matrix incorporating the social network information. La = D - A where D is a diagonal matrix whose diagonal elements Dii = ∑j Aij. Here A is the adjacency matrix of the social networks with binary entries indicating the links (relations) between items. Ajj ′ = 1 indicates that there is a link between item j and item j ′ . Otherwise, Ajj ′ = 0. PoG(θj , sj , λ-1 v IK, λ-1 r IK) in (3) denotes the product of the Gaussian N(θj , λ-1 v IK) and the Gaussian N (sj , λ-1 r IK), which is also a Gaussian [Gales and Airey, 2006]. The resulting Gaussian is N(μvr, λ-1 vr IK) with

As shown in (2) and Figure 2, the social network information is seamlessly integrated into the CTR-SR by putting the Laplacian of the adjacency matrix into the prior distribution for S. The physical meaning is to make the latent factors (sj and vj) of linked items as close as possible, which will be discussed in detail in the following content.

Since it is obviously intractable to compute the full posterior of ui, vj, sj, and θj, an EM-style algorithm is developed to learn the maximum a posteriori (MAP) estimation. We can maximize the posterior by maximizing the complete loglikelihood of U = [u1, u2, · · · , uI ], V = [v1, v2, · · · , vJ ], S, θ1:J , and R given λu, λv, λr, λl and β,

A constant is omitted and the parameter of the topic model α is set to 1 as that in CTR. Note that the first term - λl 2 tr(SLaST ) corresponds to log p(S) with a constant omitted and

where Sr∗ denotes the rth row of S and S∗c denotes the cth column of S. We can see that maximizing - λl 2 tr(ST LaS) will make sj close to sj ′ if item j and item j ′ are linked (Ajj ′ = 1).

The function L in (5) can be optimized using coordinate ascent. We first fix parameters β and optimize the collaborative filtering variables {ui, vj, sj} and the topic proportions θj iteratively. The parameter β is updated every time {ui, vj , sj} and θj are optimized.

The update rules for ui and vj are:

where Ci is a diagonal matrix with {cij, j = 1, . . . , J} as its diagonal entries and Rj is the jth row of R.

For social latent matrix S, we fix all rows of S except the kth one Sk∗ and then update Sk∗. After taking the gradient of L with respect to Sk∗ and setting it to 0, we get the following linear system:

One direct way to solve the linear system is to set Sk∗ = λr(λlLa + λrIJ)-1Vk∗. However, the complexity for one single update is O(J 3 ) where J is the number of items. Inspired by [Li and Yeung, 2009], we use the steepest descent method [Shewchuk, 1994] to iteratively update Sk∗:

As discussed in [Li and Yeung, 2009], using the steepest descent method instead of solving the linear system directly can dramatically reduce the computation cost in each iteration from O(J 3 ) to O(J).

For θj, we first define q(zjn=k) = ψjnk as that in CTR and LDA [Blei et al., 2003] and apply Jensen's inequality after items containing θj are separated,

Here φj = (φjnk)N×K n=1,k=1 . ObviouslyL(θj,φj) is a tight lower bound of L (θj) and we can use projection gradient to optimize θj. The optimal φjnk is

As for the parameter β, we follow the same M-step update as in LDA [Blei et al., 2003],

5 Experiments

We conduct experiments on two real-world data sets to demonstrate the effectiveness of our models. As stated in Section 2, although our focus is on tag recommendation for articles (papers) in this paper, our models are general enough to model other kinds of data like image tagging.

5.1 Dataset

Two real-world datasets are used in our experiments. Both of them are from CiteULike 5 , but they are collected in different ways. The first dataset, called citeulike-a, is from [Wang and Blei, 2011]. Note that there is not tag information in the original dataset of [Wang and Blei, 2011]. We collect the tag information from CiteULike. We collect the second dataset, called citeulike-t, by ourselves. Specifically, we manually select 273 seed tags and collect all the articles with at least one of these tags. Note that the final number of tags (19107 and 52946 respectively for two datasets) corresponding to all the collected articles is far more than the number of seed tags (273). We remove tags used less than 5 times and get 7386 and 8311 tags for citeulike-a and citeulike-t, respectively. There are 16980 items (articles) and 25975 items in the datasets citeulike-a and citeulike-t, respectively. The ratios of non-empty entries (equal to 1-sparsity) in the item-tag matrices of citeulike-a and citeulike-t are 0.00145 and 0.00104 respectively, which means that the second dataset is sparser than the first one.

We preprocess the text information (content of items) following the same procedure as that in [Wang and Blei, 2011]. As in [Wang and Blei, 2011], we also use the titles and abstracts of articles as content information of citeulike-t. We choose the top 20000 distinct words according to the tf-idf values as our vocabulary after removing the stop-words.

Because citation information is not provided in CiteULike, we use the user-article information which is available in CiteULike to construct the social networks between items. For each dataset, we construct the social network with a threshold of 4 using the user-article matrices. More specifically, if two items have 4 or more users in common, they are linked in the social network. This is meaningful because two papers with similar users (readers) typically have similar topics. We then merge this social network and the citation network between papers to get the final network. After network constructing, the numbers of links in the final networks are 294072 and 180103 for citeulike-a and citeulike-t, respectively.

5.2 Evaluation Scheme

In each dataset, we randomly select P items associated with each tag to construct the training set and use all the rest of the dataset as test set. We vary P from 1 to 10 in our experiments and the smaller P is, the sparser the training set is. Note that when P = 1, only 4.1% of the tagging entries are put in training set for dataset citeulike-a and the number is 3.7% for dataset citeulike-t. For each P we repeat the evaluation 5 times with randomly selected training set, and the average performance will be reported.

As in [Wang and Blei, 2011] and [Marinho and SchmidtThieme, 2007], we use recall as our evaluation metric since zero entries may be caused either by irrelevance between the tag and the item or by users who do not know the existence of the tags when tagging items, which means precision is not a proper metric here. Like most recommender systems, we sort the predicted ratings of candidate tags and recommend the top M tags to the target item. For each item, recall@M is defined as

5 CiteULike allows users to create their own collections of articles. There are abstracts, titles, and tags for each article. Other information like authors, groups, posting time, and keywords is not used in this paper. The detailed information can be found at http://www.citeulike.ort/faq/data.adp

The final reported result is the average of all the items' recall. Besides, as in [Sigurbj¨ornsson and van Zwol, 2008], we use success@M to be another evaluation metric. success@M is defined as the probability of finding a true tag among the top M recommended tags.

5.3 Baselines and Experimental Settings

We use the following baselines for comparison:

• TAGCO: This method belongs to the category of cooccurrence based methods, which is described in [Sigurbj¨ornsson and van Zwol, 2008].
• SCF: This is a similarity-based collaborative filtering [Marinho and Schmidt-Thieme, 2007]. It finds k nearest neighbors of the paper's existing tags and recommends other tags according to its neighbors' tags. It only uses the item-tag matrix information.
• CF: This is a matrix factorization based collaborative filtering [Koren et al., 2009] method. It factorizes the training matrix into two low-rank matrices U, V , and recovers the original matrix by UV T . It only uses the item-tag matrix information.
• SCF+LDA: This method integrates similarity-based collaborative filtering with LDA. It falls into the category of hybrid methods and is adapted from [Sevil et al., 2010].
• CTR: The method introduced in Section 3.

We use a validation set to find the optimal parameters. More specifically, we find that CTR achieves good prediction performance when λv = 10, λu = 0.1, a = 1, b = 0.01, and K = 200. For CF, the parameters are λv = 1, λu = 1, a = 1, b = 0.01, and K = 200. For CTR-SR, the parameters are λv = 10, λu = 0.1, a = 1, b = 0.01, K = 200, λr = 100 and λl = 10.

5.4 Performance

Figure 3 (a) and Figure 4 (a) show the recall@50 when P is set to be 1, 2, 5, 8, 10, on citeulike-a and citeulike-t, respectively. The random baselines are 0.68% and 0.60% respectively. As we can see, the hybrid method SCF+LDA outperforms those methods use only one kind of information, and CTR outperforms SCF+LDA. Furthermore, our CTR-SR model achieves the best performance for most cases by effectively integrating the social networks between items.

Figure 3 (b) and Figure 4 (b) show the recall of all the methods when P is fixed to be 5 by setting M=2, 5, 10, 20, 50 in dataset citeulike-a and citeulike-t. Figure 3 (c) and Figure 4 (c) show the success@M of all the methods when P is fixed to be 5 by setting M=2, 5, 10, 20, 50 in two datasets. Once again, we can see that CTR outperforms other baselines and CTR-SR is significantly better than other methods for most cases. Similar phenomena are observed for other values of P , which are omitted due to space constraint.

5.5 Sensitivity to Parameters

Figure 5 (a) shows how the prediction performance of CTR-SR is affected by the parameter λl. P is set to 5, λv = 10, λu = 0.1, and λr = 100. As we can see, the prediction performance first increases with λl and starts to slightly decrease at some point after λl = 10 for all values of M. It is not too sensitive in a large range of values.

Figure 5 (b) demonstrates the sensitivity of CTR-SR to parameter λr. In this experiment, P is also set to 5 and λv = 10, λu = 0.1, and λl = 10. As the figure shows, the performance first increases with λr and begins to decrease at some point after λr = 100 for all values of M. It is also not too sensitive in a large range of values.

5.6 Interpretability

Besides promising prediction performance, our proposed model can also provide a very good interpretation. Two example articles (items) with their top 2 topics are presented in Table 1. Note that although the learned topic proportions of CTR are different from those of CTR-SR, the ranking of top 2 topics are the same. In this case study, CTR-SR and CTR are trained using the extremely sparse training data (P = 1) and recommend tags to articles. Note that in the training data, each tag is associated with only one single article, which makes tag recommendation very challenging. As we can see from Table 1, for Article I, precisions of the top 10 tags for CTR-SR and CTR are 50% and 10%, respectively. For Article II, the precisions are 60% and 10%, respectively. We can find that the social network information among items are very informative and our CTR-SR model can effectively exploit it.

When examining more closely, we can find that Article I 'How much can behavioral targeting help online advertising?' is about online advertising, which can also be verified by the true tags shown in the table. As we can see, the recommended tags by CTR focuses more on the technical details while those returned by CTR-SR are closer to the essence of the articles. Similarly, Article II 'Lowcost multitouch sensing through frustrated total internal reflection' focuses on multitouch sensing. Tags recommended by CTR like 'nanoparticles', 'dna-nanotechnology', and 'gamma' seem a lot more technical and achieve a low precision of 10%. On the contrary, tags recommended by CTR-SR like 'multi-touch' and 'screen' can better describe the main points of the article and achieve a high precision of 60%.

6 Conclusion

In this paper, we first adapt CTR to combine both item-tag matrix and item content information for tag recommendation. Furthermore, by extending CTR we propose a novel hierarchical Bayesian model, called CTR with social regularization (CTR-SR), to seamlessly integrate the item-tag matrix, item content information, and social networks between items into the same principled model. Experiments on realworld datasets successfully demonstrate the effectiveness of our proposed models.

7 Acknowledgements

This work is supported by the NSFC (No. 61100125), the 863 Program of China (No. 2012AA011003), and the Program for Changjiang Scholars and Innovative Research Team in University of China (IRT1158, PCSIRT).