Recommendation with Multi-Source Heterogeneous Information

1 Introduction

With the massive amount of data generated by online social services, recommender systems are playing an important role in connecting users and information resources. To tackle the sparsity problem of user-item interactions, hybrid recommendation methods which combine collaborative filtering and auxiliary information sources such as item contents have shown promising results [Wang and Blei, 2011; Zhang et al., 2016; Gao et al., 2017; Yamasaki et al., 2017; Dong et al., 2017]. These methods focus on extracting a set of important factors for items obtained from auxiliary information.

Recently, network embedding [Perozzi et al., 2014; Chang et al., 2015; Zhang et al., 2017; Liu et al., 2018] has gained increasing popularity in social network recommendations. Network embedding aims to learn a vector representation of each node by mapping it into a low-dimensional vector space while preserving its neighborhood relationship. Because network embedding can capture the neighborhood similarity and community membership, it has been popularly used in recommendations [Chen et al., 2015; Zhao et al., 2016]. For example, the work [Zhao et al., 2016] learns the network representation of each node in the built k-partite adoption network. The recommendation task is considered as a similarity evaluation problem by ranking the cosine similarity between user and item representations.

However, previous studies on network embedding for recommendations suffer from two shortcomings. First, they do not fully use the item information. The contextual information of items are often ignored, which leads to a shallow representation of the network. Second, network embedding and item recommendations are learned independently and their interactions are often ignored.

To address the above shortcomings, we integrate deep network representations of items with collaborative filtering for recommendation. Item information are combined from multiple heterogeneous information sources, such as item structure, textual content and tag information. We design a new deep network embedding component by using two coupled neural networks which can extract deep representations from multiple heterogeneous information sources. To combine collaborative filtering and network representations obtained from multiple sources, we present a new Collaborative multisource Deep Network Embedding method (CDNE for short) to learn different latent representations. Figure 1 shows an illustration of CDNE for item recommendations.

The main contributions are summarized as follows:

• We present a new item recommendation framework that can embed deep network representations obtained from multiple information sources such as structure, textual content and tag information for item recommendations.
• We develop a new method that jointly performs multi-

source deep network embedding and collaborative filtering, where deep representations of items and interactions between users and items are learned collaboratively.

• We compare our method with state-of-the-art methods on two real-world data sets to evaluate the performance. Experimental results demonstrate that our method significantly outperforms the baseline methods in terms of the precision and MRR metrics.

2 Problem Statement

We denote users by U = {u1, u2, . . . , um}, and items by V = {v1, v2, . . . , vn}. The browsing history of users is recorded by an m × n user-item interaction matrix R. Rij is either a rating score given by ui on vj or a missing value (Rij = 0). We consider item information from multiple sources: item structure, textual content and tag information. Given the above item information and the history of interactions between users and items, we aim to recommend each user ui with a ranked list of interested items.

Since we explore to enhance the quality of recommendation by leveraging deep network representations, each item vj is mapped to a node of network G = (V, E,D, C), where eij ∈ E denotes the edge relationship 1 from vi to vj, dj ∈ D denotes the textual content associated with item vj, and cj ∈ C denotes the tag information associated with item vj .

We assume that the connected items are statistically dependent, which learns the inter-item relationship from the random walk corpus generated from item structure E. The textual content D captures the item-content correlation and the tag information C represents the tag-content correspondence between item tag and item content.

To obtain deep network representations from multiple sources, we develop a deep network embedding component by utilizing two coupled neural networks, where the deep representation vector θj of item vj and the latent representation vector lj of cj are taken as input. The output is the textual word vectors and the representation vectors of nodes in the contextual window. In the collaborative item recommendation component, the latent factor vector V j is formulated as a combination of θj and the latent offset vector ϵj. We then combine V j and user latent factor vector Ui for joint learning of user-item interactions.

1 Without loss of generality, we assume that G is a directed graph. The case of undirected networks can be readily adapted by replacing each undirected edge with two oppositely directed edges.

3 Preliminary: DeepWalkModel

Based on the Skip-gram model [Mikolov et al., 2013a], DeepWalk [Perozzi et al., 2014] constructs a corpus S that consists of random walks generated from the network. Each random walk s = {v1, . . . , vns} is considered as a sentence and each node vj is regarded as a word in neural language models. Assume that aj = {vj-c, . . . , vj+c}\vj is the context vertices when given the target node vj. DeepWalk aims to maximize the following objective

Note that DeepWalk only utilizes the network structure information for model learning.

4 Our Solution

In this section, we introduce our proposed method CDNE for recommendation, which integrates collaborative filtering with deep network representations of items.

4.1 Deep Network Embedding

Network embedding provides an effective way to capture neighborhood similarity and community membership, which is beneficial for recommendation [Zhang et al., 2017]. We map each item vj to a node of network G. We adapt DeepWalk model to learn the deep representation vector θj of vj from not only the network structure information, but also textual content and tag information augmented with each item.

For the random walk sequence generation [Pan et al., 2016], we take network structure as input to construct corpus S. Each walk sequence samples uniformly a random node vj as the root and randomly jumps to one node chosen from the neighbors of the last node visited.

For item vj , we assume that 1) the deep representation vector θj is influenced by the random walk sequences that have visited vj, the textual content dj, and the tag information cj; and 2) cj also specifies the words in dj, which models the correspondence between item tags and item content. To be specific, as illustrated in Figure 1, we couple two neural networks by the item vj, indicating that θj acts as the input for both the two neural networks.

The first neural network models the generated random walk sequences, for which the input is θj and the output is the deep representations of its context items aj = {vj-c, . . . , vj+c}\vj. The objective function can be formulated as follows

The second neural network models the textual content dj, for which the input is θj and lj, and the output is the latent vectors of words in dj . lj denotes the latent tag representation vector for cj . Assume that wj = {wj1 , . . . , wj2c } is a sequence of textual words within a contextual window. The objective aims to maximize the following likelihood function

Note that the first term is similar to the paragraph vector model [Le and Mikolov, 2014] that learns the latent representation for each document from the textual information.

Combining the above two objectives in Eq. (2) and Eq. (3), our coupled neural networks aim to maximize the following objective function

where σ′s and σ′t are used to balance the weights of item structure, textual content and tag information.

As suggested by the previous work [Mikolov et al., 2013b], the probability P(vk | vj) in Eq. (2) can be calculated by using the softmax function as follows

where ovj and o′vj are the input and output vector representation of item vj. P(wk | vj) and P(wk | cj) in Eq. (3) can be readily calculated using softmax function as in Eq. (5). After training the neural network model, the input vector ovj can be used as the deep representation vector θj of vj .

4.2 Collaborative ItemRecommendation

Most successful collaborative filtering recommendation methods are latent factor models, among which matrix factorization performs well [Salakhutdinov and Mnih, 2007; Gao et al., 2016]. We represent users and items in a shared latent low-dimensional space of dimension K, where user ui is represented by a latent factor vector Ui ∈ RK and item vj by a latent factor vector V j ∈ RK . The user-item interactions can be formulated as

where the variable σij serves as a confidence parameter for rating Rij. we set σij = a, if ui has rated vj; otherwise, σij = b, where a and b are tuning parameters satisfying a > b > 0. A similar strategy is used in [Wang and Blei, 2011].

We introduce a latent variable ϵj ∈ RK , ϵj ∼ N(0, σ-1 v I), to offset the deep network embedding θj when modeling the historical user-item interactions. To collaboratively capture an item's latent deep representation frommulti-source items' information and latent factor vector in collaborative filtering, the item latent vector V j is formulated as

The generative process of CDNE that recommends items with multi-source deep network embedding is described as follows

1. For user ui, draw a latent factor vector Ui ∼ N (0, σ-1 u I),
2. Considering deep network embedding learned from multiple sources:
3. (a) Given item vj, for the random walk sequence s ∈ S, draw from the probability P(aj | vj),
4. (b) For each item vj with its textual content dj, draw from the probability P(wj | vj),
5. (c) For each item vj with its tag information cj, draw from the probability P(wj | cj).
6. For item vj, draw an item latent offset ϵj ∼ N (0, σ-1 v I) and set V j = θj + ϵj,
7. For each user-item pair (ui, vj), draw the rating Rij ∼ N(U Ti V j, σ-1 ij ).

Based on the above steps, computing the full posterior of the parameters is intractable. As suggested by the previous work [Wang and Blei, 2011], maximizing the posterior probability of U, V , θ and l is equivalent to minimizing the complete negative log-likelihood as follows

where σij, σs, σu, σv and σt are the weight parameters.

4.3 Parameter Optimization

We use stochastic gradient descent to solve the objective in Eq. (8). As shown in Eq. (5), the probability P(vk | vj) (P(wk | vj) or P(wk | cj) are the same) is calculated by the softmax function. To reduce the computation cost of the gradient of P(vk | vj), P(wk | vj) or P(wk | cj), we instead use the hierarchical softmax [Morin and Bengio, 2005] to approximate the probability distribution.

Specifically, for the calculation of P(vk | vj), we assign distinct nodes as leaves of a binary tree, which is built using the Huffman coding [Mikolov et al., 2013b] to assign shorter paths to the frequent nodes in random walks. While for P(wk | vj) or P(wk | cj), we assign the distinct words as leaves of another binary tree. Then, there is a unique path from the root to each leaf. Suppose the path to node vj is identified by a sequence of tree nodes, f0, f1, . . . , fm, P(vk | vj) is then calculated by the probability of the specific path. We have

where P(ft | vj) is defined as P(ft | vj) = 1/(1+e-oTvj o′ft ). o′ft ∈ RK denotes the representation assigned to tree node ft's parent. Similarly, as in Eq. (9), we can use hierarchy softmax technique to calculate P(wk | vj) or P(wk | cj).

Substituting for P(vk | vj), P(wk | vj) and P(wk | cj) in Eq. (8) using hierarchy softmax, we denote L as the objective. The model parameter set becomes Θ = {U, V , θ, l, Ψ}, where Ψ denotes the representations assigned to the interval nodes of the binary tree. In each iteration, we use the gradient to update each parameter in Θ as follows

where αt is the learning rate. Due to space limitations, we omit the concrete formulations of parameters update.

4.4 Prediction

Given the history of user-item interactions, as well as the multi-source heterogeneous information of items, we obtain the optimal parameters after solving the objective function in Eq. (8). We then recommend each user ui with a list of items, v1, v2, . . . , vh, using the ranking criterion U Ti V 1 ≥ U Ti V 2 ≥ · · · ≥ U Ti V h.

5 Experiments

5.1 Data Sets

We use two real-world data sets [Wang et al., 2013] extracted fromCiteULike 2 for experimental analysis.

The first data set, called citeulike-a, contains 16,980 items (i.e., articles) and 7,386 tags for the items. The second data set, called citeulike-t, contains 8,311 tags and 25,975 items. Following the work [Wang et al., 2013], for each article in the data sets, we use its title and abstract as the textual content. We remove stop words and use tf-idf to choose the top 20,000 distinct words as the vocabulary. Because citation information is not provided in CiteULike, to construct social networks between items, we use the user-article information following the same procedure as in [Wang et al., 2013]. For each data set, if two items share more than four users, they are linked in the social network. This is because two articles with similar users typically have similar topics. The items' network structure then contains 294,072 links and 180,103 for citeulike-a and citeulike-t data set, respectively.

5.2 Experimental Settings

Baselines

We compare our method CDNE with the following benchmark methods: 1)PMF [Salakhutdinov and Mnih, 2007] is an effective probabilistic matrix factorization method for recommendation; 2) CTR [Wang and Blei, 2011] combines traditional collaborative filtering with topic modeling for recommendation; 3) SLG [Chen et al., 2015] proposes music recommendation method (we denote it as SLG and adapt it for our recommendation tasks) by integrating the network representations into factorization machines; 4) NERM [Zhao et al., 2016] recommends items by ranking the cosine similarity between representations of users and items obtained from the bipartite adoption graph; 5) CDNE-st is a variant method of CDNE, which considers multi-source items' information excluding items' network structure; 6) CDNE-tc is a variant that excludes items' textual content; 7) CDNE-ta is a variant that excludes items' tag information.

2 The detailed information of the data can be found at http://www.citeulike.org/faq/data.adp. CiteULike allows users to create their own collections of articles. There are titles, abstracts and tags for each article. Other information about the article, such as authors, publications and keywords, is not used in this paper.

Evaluation Metrics

Two metrics, Precision@n (P@n) and Mean Reciprocal Rank (MRR), are used to measure the performance of item recommendations. P@n measures the ratio of successfully recommended items to the top-k recommendations and MRR measures the reciprocal of the first occurrence position of the ground truth item for each user [Liu, 2015]. The two metrics are first calculated separately on each user's recommendation list and then are averaged among all the test users. The higher values of the two metrics are favored in comparisons.

Settings

We randomly partition each of the two data sets into training and testing sets. For each user ui, 70% of the items (i.e., articles) are randomly sampled as the training data, and the remaining 30% are used for testing. We then randomly choose one record of each user from the training data set to construct the validation data. All compared methods use the same number of latent factors K, K = 200. For all neural network models, the window size c is set as c = 8. The P@10 performance on the validation data for each data set is used to select the optimal parameters. As a result, we set the hyperparameters as a = 1, b = 0.01, σs = 1, σt = 0.5, σu = 0.1, σv = 1. The learning rate α is set as α = 0.01. For each model, we run the experiments 100 times and report the averaged results.

5.3 Experimental Results

Figure 2 shows the experimental results of the precision and MRR metrics on citeulike-a and citeulike-t data sets with respect to a range of recommendation list sizes. From the figure, we see that the SLG and NERM methods outperform the basic PMF method for both the data sets. For example, for the citeulike-a data set, the method SLG averagely improves the precision and MRR by 28.44% and 15.48%, and the method NERM improves by 45.97% and 24.30%. The above results demonstrate that the proper integration of network representations learned from the auxiliary information can boost the recommendation performance. As the representative recommendation method for textual content analysis, even though no network embedding is considered, the method CTR outperforms the methods SLG and NERM for citeulike-a (over the range of recommendation list size) and citeulike-t (when the recommendation list size is larger than 7) data sets under both the precision and MRR metrics.

We also show the experimental results of variants (CDNEst, CDNE-tc and CDNE-ta) of our method CDNE. The method CDNE-st performs the best among the three variants for both the data sets. For example, for the citeulike-a data set, compared with the CDNE-tc and CDNE-ta methods, CDNE-st averagely improves the precision by 38.36% and 16.85%, and the MRR by 8.39% and 4.45%. The reason may be that the interested topics have more influence on the selection of articles for a user, where the topics can be clearly mined from the articles' textual content and tag content, such as the title, abstract and research area (e.g., artificial intelligence) the article belongs to. CDNE-tc and CDNE-ta outperforms the CTR method for all the evaluations except for the case that is measured by P@15 on the citeulike-t data set.

In all cases, our method CDNE significantly outperforms the baselines. Compared with SLG, instead of finding possible random walk path based on the user-item matrix, we combine items' deep network representations, items' offset vectors and users' latent factor vectors for jointly learning of user-item interactions. The random walks are generated on the network structure (citation information among articles) to learn network representations from items' multi-source information. The method NERM aims to separately learn network embedding on the bipartite adoption graph and make item recommendations with these embeddings, ignoring the interactions between network representation and the objective of recommendation. We instead propose a unified framework to learn different latent factor vectors collaboratively. In this way, we can properly capture their interactions.

Table 1 summarizes the precision and MRR performance of all the compared methods averaged over different recommendation list sizes, which shows the similar results as the above. In summary, by averaging the performance when different recommendation list sizes and different data sets are applied, our method CDNE improves the methods PMF, CTR, SLG, NERM, CDNE-st, CDNE-tc and CDNE-ta by 156.42%, 62.76%, 95.28%, 80.35%, 20.87%, 49.65% and 33.33% in terms of the precision metric, and by 87.92%, 40.42%, 61.32%, 54.82%, 20.03%, 30.31% and 25.00% in terms of the MRR metric.

5.4 Case Study

Following the work [Wang et al., 2013], two articles linked in the social network typically have similar topics. The random walks are generated on the citation graph, where the items'

network embeddings are determined by item structure, textual content and tag information. From this point of view, the network embedding can be represented as a latent topic distribution, and users are assumed to have topic interests. Then, we can recommend articles to the users using the latent topic distribution and topic interests. To give a clear illustration of the recommendation performance, Table 2 shows the top5 recommendation results on the citeulike-t data set for an example user (userID: 2975), when the user creates her own collections of the article (#6774). All the compared methods predict the results that the example user may be interested in. Our method CDNE gives the best results.

6 RelatedWork

Our work relates to the research area of collaborative filtering and network representation learning.

Collaborative filtering based methods [Salakhutdinov and Mnih, 2007] use historical interactions or preferences to recommend items. However, due to the issues of sparsity of useritem interactions, collaborative filtering based models usually suffer from the limited performance. By using auxiliary information, hybrid recommender models [Wang and Blei, 2011; Qiao et al., 2014; Zhang et al., 2016; Gao et al., 2017; Dong et al., 2017] usually obtain better recommendation results. The work [Wang and Blei, 2011] proposed a collaborative topic regression model which combines traditional collaborative filtering methods with topic modeling. A hybrid model [Dong et al., 2017] was proposed to jointly learn deep users and items latent factors from side information and collaborative filtering from the rating matrix.

Hybrid recommendation models aim to learn latent factors of users and items from user-item interactions and auxiliary information, which motivates our work in this paper. On the other hand, network representation learning [Chang et al., 2015; Wang et al., 2016; Ribeiro et al., 2017] attracts increased attention. This type of method aims to learn Article's title: Towards the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions

CDNE:

PMF:

1. Item-based collaborative filtering recommendation algorithms
2. Effective missing data prediction for collaborative filtering
3. Toward trustworthy recommender systems: An analysis of attack 2. GroupLens: applying collaborative filtering to Usenet news models and algorithm robustness
4. Information Filtering: Overview of Issues, Research and Systems
5. Incorporating contextual information in recommender systems using a multidimensional approach
6. Factorization meets the neighborhood: a multifaceted collaborative filtering model
7. Content-based book recommending using learning for text categorization
8. An efficient boosting algorithm for combining preferences
9. Incorporating contextual information in recommender systems using a multidimensional approach

CTR:

SLG:

1. Toward trustworthy recommender systems: An analysis of attack 1. Content-based book recommending using learning for models and algorithm robustness text categorization
2. Analysis of recommendation algorithms for e-commerce
3. Item-based collaborative filtering recommendation algorithms
4. Effective missing data prediction for collaborative filtering
5. Information Filtering: Overview of Issues, Research and Systems
6. GroupLens: applying collaborative filtering to Usenet news
7. Amazon.com recommendations: item-to-item collaborative filtering
8. Analysis of recommendation algorithms for e-commerce
9. Methods and metrics for cold-start recommendations

NERM:

CDNE-st:

1. Item-based collaborative filtering recommendation algorithms
2. Item-based collaborative filtering recommendation algorithms
3. An efficient boosting algorithm for combining preferences
4. Effective missing data prediction for collaborative filtering
5. Information Filtering: Overview of Issues, Research and Systems
6. Effective missing data prediction for collaborative filtering
7. How oversight improves member-maintained communities
8. Analysis of recommendation algorithms for e-commerce
9. Amazon.com recommendations: item-to-item collaborative filtering
10. How much can behavioral targeting help online advertising?

CDNE-tc:

CDNE-ta:

1. Toward trustworthy recommender systems: An analysis of attack 1. Methods and metrics for cold-start recommendations models and algorithm robustness
2. Factorization meets the neighborhood: a multifaceted collaborative filtering model
3. GroupLens: applying collaborative filtering to Usenet news
4. Effective missing data prediction for collaborative filtering
5. Item-based collaborative filtering recommendation algorithms
6. Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness
7. Item-based collaborative filtering recommendation algorithms
8. Information Filtering: Overview of Issues, Research and Systems
9. Large-scale behavioral targeting

Table 2: Top-5 recommendation results on the citeulike-t data set for an example user (userID: 2975) when the user creates her own collections of the article (#6774). The number in bold indicates that the corresponding article is correctly predicted.

low-dimensional latent representations of nodes in networks. These methods are widely used in network classification [Perozzi et al., 2014], network visualization [Tang et al., 2015b], link prediction [Grover and Leskovec, 2016], text classification [Tang et al., 2015a] and community detection [Wang et al., 2017].

However, using network representation models for item recommendations has not been fully studied. In this paper, we aim to solve the recommendation problem based on multisource network representation methods. The work [Chen et al., 2015] proposes to integrate network representations learned from the DeepWalk model into the factorization machines and obtains better recommendation results. By using the network embedding, the work [Zhao et al., 2016] treats the recommendation problem as a cosine similarity calculation between representations of users and items.

In fact, previous studies only use network structure for representation learning, ignoring other types of auxiliary information. They also separate the learning of network embedding and recommendation into two independent steps. Zhang et al. [Zhang et al., 2017] leverage the heterogeneous information to improve the recommendation results. However, the types of multi-source item information used in our work are different from their study. We further consider the tag information of items to capture the tag-content correspondence between item tag and item content. The work [Zhang et al., 2016] maps items to entries in the knowledge base, using existing deep learning method to learn item semantics, while we map items to nodes in the network using network representation learning based on the DeepWalk model.

7 Conclusion

In this paper, we develop a new network representation learning for item recommendations. We consider multiple information sources, i.e., item structure, textual content and tag information and present a new method CDNE that integrates collaborative filtering with deep network representations of items for recommendation. Compared with the baseline methods, CDNE obtains the better experimental results in terms of the Precision and MRR metrics. Therefore, by exploiting deep network embedding of items obtained from multi-source heterogeneous information and user-item interactions, our method can be used to boost the recommendation performance.

Acknowledgments

We would like to thank the anonymous reviewers for their valuable comments and suggestions. This work was supported by the National Key Research and Development Program of China (No.2017YFB0803300), the NSFC (No.61502479), the MQNS Grant (No.9201701203), the MQ Enterprise Partnership Scheme Pilot Res Grant (No. 9201701455), and the Youth Innovation Promotion Association CAS (No. 2017210). C. Zhou is the corresponding author.