A Graphical and Attentional Framework for Dual-Target Cross-Domain Recommendation

1 Introduction

Targeting data sparsity problem, Cross-Domain Recommendation (CDR) [Berkovsky et al., 2007] aims to leverage the richer information from a richer domain to only help improve the recommendation accuracy on a sparser domain, resulting in single-target CDR. In contrast, the novel dual-target CDR has been recently proposed to improve the recommendation accuracies on both richer and sparser domains simultaneously by making good use of the information or knowledge from both domains [Zhu et al., 2019; Li and Tuzhilin, 2019].

Intuitively, based on the existing single-target CDR approaches, it seems to be a solution for dual-target CDR by simply changing their transfer direction from

∗Contact Author

"Richer→Sparser" to "Sparser→Richer". However, as referred to as Negative Transfer [Pan and Yang, 2009], this idea does not work, because, in principle, the knowledge learned from the sparser domain is less accurate than that learned from the richer domain, and thus the recommendation accuracy on the richer domain is more likely to decline by simply and directly changing the transfer direction. Therefore, dualtarget CDR demands novel and effective solutions.

To achieve good-performing dual-target CDR, there are two significant challenges in the literature as follows.

CH1. how to leverage the data richness and diversity to generate more representative single-domain user and item embeddings for improving recommendation accuracy on each of both domains? Both traditional Collaborative Filtering (CF) models, e.g., BPR [Rendle et al., 2009], and novel neural CF models, e.g., NeuMF [He et al., 2017] and DMF [Xue et al., 2017], are based on the user-item relationship to learn user and item embeddings. However, most of them ignore the user-user and item-item relationships, and thus can hardly enhance the quality of embeddings.

CH2. how to effectively optimize the user or item embeddings on each target domain for improving recommendation accuracies on both domains? The state-of-the-art dual-target CDR approaches either adopt fixed combination strategies, e.g., average-pooling, max-pooling, and concatenation [Zhu et al., 2019], or simply adapt the existing single-target transfer learning to dual transfer learning [Li and Tuzhilin, 2019]. However, none of them can effectively combine the embeddings of common users/items, and thus it is hard to achieve an effective embedding optimization on each target domain.

Our Approach and Contributions. For the above two challenges, in this paper, we propose a novel graphical and attentional approach for dual-target CDR. The characteristics and contributions of our work are summarized as follows:

• We propose a Graphical and Attentional framework for Dual-Target Cross-Domain Recommendation, called GADTCDR, which can leverage the data richness and diversity (e.g., ratings, reviews, and tags) of dual domains, share the knowledge of common users across domains, and make recommendations on both domains;
• To address CH1, we construct a heterogeneous graph, considering not only user-item relationships (based on ratings),

but also user-user and item-item relationships (based on content similarities). Then, with this heterogeneous graph, we apply a graph embedding technique, i.e., Node2vec, to generate more representative single-domain user and item embeddings for accurately capturing user and item features;

• To address CH2, we propose an element-wise attention mechanism to effectively combine the embeddings of common users learned from both domains. This mechanism trains two separate element-wise attention networks for the two target domains respectively, which can significantly enhance the quality of user embeddings and thus improve the recommendation accuracy on each of two domains simultaneously;
• We conduct extensive experiments on four real-world datasets, which demonstrate that our GA-DTCDR significantly outperforms the best-performing baselines by an average of 8.46% in terms of recommendation accuracy.

2 RelatedWork

2.1 Single-Target and Dual-Target CDR

Single-Target CDR. According to transfer strategies, the existing single-target CDR approaches can be generally classified in two categories, i.e., content-based transfer and feature-based transfer. Content-based Transfer approaches first link the richer and sparser domains by content information, such as attributes [Berkovsky et al., 2007] and text [Tan et al., 2014], then share user/item knowledge across domains. Feature-based Transfer approaches tend to apply some classical machine learning techniques, e.g., multi-task learning [Lu et al., 2018], transfer learning [Hu et al., 2019], and neural networks [Man et al., 2017; Zhu et al., 2018; Fu et al., 2019], to directly share or indirectly map user/item embeddings or rating patterns [Yuan et al., 2019] across domains. Also, some studies [Pan and Yang, 2013; Zhang et al., 2016] focus on a derivational problem, i.e., multi-domain recommendation, which is to improve the recommendation accuracy on the target domain by leveraging the auxiliary information from multiple domains. However, all of them are single-target models, which means they cannot improve the recommendation accuracy on the richer domain even if the sparser domain may contain certain types of auxiliary information to support the richer domain.

Dual-Target CDR. The existing dual-target CDR approaches mainly focus on applying fixed combination strategies, e.g., average-pooling, max-pooling, and concatenation [Zhu et al., 2019], or simply improving the existing singletarget transfer learning to be a dual transfer learning [Li and Tuzhilin, 2019]. However, none of them can effectively combine the embeddings of common users.

2.2 Graph Embedding

Graph Embedding is to learn a mapping function which maps the nodes in a graph to low-dimensional latent representations [Zhou et al., 2018; Liu et al., 2019]. According to embedding techniques, we review the existing graph embedding approaches into two categories, i.e., dimensionality reduction and neural network. Dimensionality reduction-based approaches [Wold et al., 1987] optimize a function that reduces the dimension of a graph matrix and then produce lowdimensional embeddings. Neural network-based approaches, e.g., Node2vec [Grover and Leskovec, 2016], treat nodes as words and the generated random walks on graphs as sentences, and then learn node embeddings based on these words and sentences.

2.3 AttentionMechanism

Attention is firstly introduced in [Bahdanau et al., 2014], which provides more accurate alignment for each position in a machine translation task. Apart from machine translation, recently, attention mechanism also has been widely used in recommendation [Chen et al., 2017]. The general idea of attention mechanism is to focus on selective parts of the whole information, which can capture the outstanding features of objects. For recommendation, the existing attention approaches [Wang et al., 2019] tend to select more informative parts of explicit or implicit data to improve the representations for users and items.

3 The ProposedModel

In this section, we first formalize the dual-target cross-domain recommendation problem. Then, we propose a novel Graphical and Attentional framework for Dual-Target Cross-Domain Recommendation, called GA-DTCDR. Finally, we present the detailed components of GA-DTCDR.

3.1 Problem Statement

First, for readability purposes, we list the important notations of this paper in Table 1. Then, we define the Dual-Target Cross-Domain Recommendation as follows.

Dual-Target Cross-Domain Recommendation (DTCDR). Given two related domains a and b including explicit feedback (e.g., ratings and comments), implicit feedback (e.g., purchase and browsing histories), and side information (e.g., user profiles and item details), the DTCDR is to improve the recommendation accuracies on both domains simultaneously by leveraging their observed information.

Note that a certain degree of overlap between the users of domains a and b, i.e., common users, plays a key role in bridging the two domains and exchanging knowledge across domains. This is a common idea in the existing single-target and dual-target CDR approaches.

3.2 Overview of GA-DTCDR

Targeting the dual-target CDR problem, we propose a novel graphical and attentional framework, called GA-DTCDR. As shown in Fig. 1, this framework is divided into five main components, i.e., Input Layer, Graph Embedding Layer, Feature Combination Layer, Neural Network Layers, and Output Layer. We will present the details of each component below.

Input Layer. First, for the input of our GA-DTCDR, we consider both explicit feedback (ratings and comments) and side information (user profiles and item details). These input data can be generally classified into two categories, i.e., rating information and content information.

Graph Embedding Layer. Then, we leverage the rating and content information of domains a and b to construct a heterogeneous graph representing user-item interaction relationships, user-user similarity relationships, and item-item similarity relationships. Based on the graph, we apply the Graph Embedding model, Node2vec [Grover and Leskovec, 2016], to generate user and item embedding matrices.

Feature Combination Layer. Next, we propose an element-wise attention mechanism to combine the common users' embeddings for domains a and b. This layer intelligently gives a set of weights to the two embeddings of a common user learned from both domains and generates a combined embedding for the common user, which remains his/her features learned from domains a and b with different proportions.

Neural Network Layers. In this component, we apply a fully-connected neural network, i.e., Multi-Layer Perceptrons (MLP), to represent a non-linear relationship between users and items on each domain.

Output Layer. Finally, we can generate final user-item interaction predictions. The training of our model is mainly based on the loss between predicted user-item interactions and observed user-item interactions.

In fact, like the single-target or dual-target CDR approaches in [Zhao et al., 2017; Zhu et al., 2018; Zhu et al., 2019], our GA-DTCDR framework can be applied to Cross-System Recommendation (CSR) as well, where the two systems have the same domain but different users, and thus contain common items only, such as DoubanMovie and MovieLens (see Task 3 in Experiments and Analysis). Accordingly, in Fig. 1, we only need to replace common users with common items for supporting dual-target CSR.

3.3 Graph Embedding Layer

The existing embedding strategies for recommender systems mainly focus on representing the user-item interaction relationship. Apart from the user-item interaction relationship, we use a graph to represent user-user and item-item relationships as well. Therefore, based on the rating and content information observed from domains a and b, we construct two heterogeneous graphs, including nodes (users and items) and weighted edges (ratings and content similarities), for domains a and b, respectively. Then, we can generate more representative user and item embedding matrices. The Graph Embedding contains three main components, i.e., Document Embedding, Graph Construction, and Output.

Document Embedding. To construct the heterogeneous graph, we need to compute the content similarities between two users or two items. To this end, we consider multi-source content information, e.g., reviews, tags, user profiles, item details, observed from domains a and b, to generate user and item content embedding matrices. In this paper, we adopt the most widely used model, i.e., Doc2vec [Le and Mikolov, 2014], as the document embedding technique. The detailed document embedding process works as follows: (1) First, in the training set, for a user ui, we collect the comments (reviews and tags) Ci∗ and the user profile upi of ui into the same document di, while for an item vj, we collect the comments (reviews and tags) C∗j on the item and its item detail idj into the same document dm+j; (2) Next, we segment the words in the documents D = {d1, d2, ..., dm+n} by using the most widely used natural language tool, i.e., StanfordCoreNLP [Manning et al., 2014]; (3) Finally, we apply Doc2vec model to map the documents D into the text vectors UC and V C for users and items, respectively.

GraphConstruction. First, we link the users and items via their interaction relationships. The weights of these interaction edges are normalized ratings, i.e., R/max(R). To consider the user-user and item-item relationships in the heterogeneous graph, we generate the synthetic edges between two users or two items according to their normalized content similarities (edge weights). The generation probability P(i, l) of the edge between users ui and ul is as follows:

where α is a hyper-parameter which controls the sampling probability and sim(UCi,UCl) is the normalized cosine similarity between UCi and UCl. Similarly, we can obtain the generation probability between two items. Based on the user-item interaction relationships, user-user similarity relationships, and item-item similarity relationships, we can construct the heterogeneous graphs Ga and Gb for domains a and b, respectively.

Output. Based on the two heterogeneous graphs Ga and Gb , we employ the graph embedding model, i.e., Node2vec [Grover and Leskovec, 2016], to generate user embedding matrix U and item embedding matrix V .

3.4 Feature Combination Layer

In this layer, we combine the embeddings of common users learned from domains a and b by an element-wise attention mechanism. By this way, the combined embeddings of common users ˜U for each domain can remain both features learned from the two domains in different proportions. The traditional attention mechanism tends to select a certain part of representative features, and gives these features higher weights when generating the combined features [Bahdanau et al., 2014]. Similarly, for a common user ui, our element-wise attention mechanism tends to pay more attention to the more informative elements from each pair of elements in U a i and U b i . Thus our element-wise attention mechanism can generate two more representative embeddings ˜U a i and ˜U b i of the common user ui for domains a and b, respectively. The structure of element-wise attention is shown in Feature Combination Layer of Fig. 1. The combined embedding ˜U a i of a common user ui for domain a can be represented as:

where ⊙ is the element-wise multiplication and W a ∈ Rma×k is the weight matrix for the attention network on domain a. Similarly, we can obtain the combined embedding ˜U b i of ui for domain b.

Note that for the distinct users and all the items on domains a and b, we just reserve their embeddings without using the attention mechanism because they do not have dual embeddings on both domains a and b.

3.5 Training for Neural Network Layers and Output Layer

First of all, we train our model with the following objective function on domain a:

where ℓ(y, ˆ y) is a loss function between an observed interaction y and its corresponding predicted interaction ˆ y (see Eq. (5)), Y a+ and Y a- denote all the observed and the unobserved user-item interactions on domain a respectively, ‖P a ‖2F + ‖Qa ‖2F is the regularizer (see Eq. (6)) and λ

is a hyper-parameter which controls the importance of the regularizer. To avoid our model over-fitted to Y + (positive instances), we randomly select a certain number of unobserved user-item interactions as negative instances, denoted by Y - sampled , to replace Y -. This training strategy has been widely used in the existing approaches [He et al., 2017].

Based on rating information, the user-item interaction yij between a user ui and an item vi can be represented as:

We choose a normalized cross-entropy loss which can be represented as:

where max(R) is the maximum rating on a domain.

As shown in Neural Network Layers of Fig. 1, our GADTCDR employs a neural network, i.e., MLP, to represent a non-linear relationship between users and items. The input embedding matrices of users and items on domain a for the MLP are P a in = [ ˜U a ; U ad ] and Qain = V a respectively, where ˜U a is the combined embedding matrix of common users for domain a, and U ad is the embedding matrix of distinct users on domain a. Therefore the embedding of user ui and item embedding of item vj in the output layer of the MLP can be represented as:

where the activation function f(∗) is ReLU, W a P1 , W a P2 ... and W aQ1 ,W aQ2 ... are the weights of multi-layer networks in different layers on domain a for P a ini and Qainj , respectively.

Finally, in Output Layer of Fig. 1, the predicted interaction ˆ yij between ui and vj on domain a is as follows:

Similarly, we can obtain the predicted interaction ˆ yb ij on domain b.

4 Experiments and Analysis

We conduct extensive experiments on four real-world datasets to answer the following key questions: Q1: How does our model outperform the state-of-the-art models (see Result 1)? Q2: How does the element-wise attention mechanism contribute to performance improvement (see Result 2)? Q3: How does the dimension k of embeddings affect the performance of our model (see Result 3)? Q4: How does our model perform on Top-N recommended lists (see Result 4)?

4.1 Experimental Settings

Experimental Datasets and Tasks. To validate the recommendation performance of our GA-DTCDR approach and baseline approaches, we choose four real-world datasets (see Table 2), i.e., three Douban subsets (DoubanBook, DoubanMusic, and DoubanMovie) [Zhu et al., 2019], and MovieLens 20M [Harper and Konstan, 2016]. For the three Douban subsets, we retain the users and items with at least 5 interactions, while for MovieLens 20M, we extract a MovieLens subset containing 10, 000 users with at least 5 interactions as well. This filtering strategy has been widely used in the existing approaches [Yuan et al., 2019; Zhu et al., 2019]. The three Douban subsets contain ratings, reviews, tags, user profiles, and item details while MovieLens contains ratings, tags, and item details. Based on these four datasets, we design 2 CDR tasks and 1 CSR task (see Table 2) to validate the recommendation performance in CDR and CSR scenarios, respectively.

Parameter Setting. For fair comparison, we optimize the parameters of our GA-DTCDR and those of the baselines. For Graph Embedding Layer in Fig. 1, we set the hyperparameters of Doc2vec and Node2vec models as suggested in [Le and Mikolov, 2014; Grover and Leskovec, 2016], and the sampling probability α as 0.05. In Neural Network Layers of Fig. 1, the structure of the layers is "k → 2k → 4k → 8k → 4k → 2k → k", the parameters of the neural network are initialized as the Gaussian distribution X ∼ N(0, 0.01). For training our GA-DTCDR, we randomly select 7 negative instances for each observed positive instance into Y - sampled , adopt Adam [Kingma and Ba, 2014] to train the neural network, and set the maximum number of training epochs to 50. The learning rate is 0.001, the regularization coefficient λ is 0.001, and the batch size is 1, 024. To answer Q3, the dimension k of the embedding varies in {8, 16, 32, 64, 128}.

Evaluation Metrics. To evaluate the recommendation performance of our GA-DTCDR approach and baseline approaches, we adopt the ranking-based evaluation strategy, i.e., leave-one-out evaluation, which has been widely used in the literature [Xue et al., 2017; Wang et al., 2019]. For each test user, we choose the latest interaction with a test item as the test interaction and randomly sample 99 unobserved interactions for the test user, and then rank the test item among the 100 items. Leave-one-out evaluation includes two main metrics, i.e., Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) [Wang et al., 2019]. HR@N is the recall rate while NDCG@N measures the specific ranking quality that assigns high scores to hits at top position ranks. Note that we only report HR@10 and NDCG@10 results in Results 13, and HR@N and NDCG@N results in Result 4.

Comparison Methods. As shown in Table 3, we compare our GA-DTCDR with seven baseline models in three groups, i.e., (1) Single-Domain Recommendation (SDR), (2) Single-Target Cross-Domain Recommendation (CDR), and (3) Dual-Target CDR. All of them are representative and/or state-of-the-art approaches for each group. Also, for ablation study, we also implement a simplified version of our GA-DTCDR, i.e., GA-DTCDR Average (replacing elementwise attention with a fixed combination strategy, i.e., averagepooling). For clear comparison, in Table 3, we list the detailed training data types, encoding strategies, embedding strategies, and transfer strategies of all the nine models implemented in the experiments.

4.2 Performance Comparison and Analysis Result 1: Performance Comparison (for Q1)

To answer Q1, we compare the performance of our GADTCDR with those of the seven baseline models. Note that for the SDR baselines, we train them on each domain and then report their performance on each domain; for the single-target CDR baselines, we train them on both domains and then only report their performance on the sparser domain; and for the dual-target CDR models, we train them on both domains and then report their performance on each domain.

Tables 4 - 6 show the experimental results in terms of HR@10 and NDCG@10 with different embedding dimensions k for Tasks 1, 2, and 3, respectively. As indicated in Tables 4 - 6, our GA-DTCDR outperforms all the SDR, single-target CDR, and dual-target CDR baselines by an average improvement of 8.46%. In particular, our GA-DTCDR improves the best-performing baselines (with results marked by * in Tables 4 - 6) by an average of 10.34% for Task 1, an average of 10.29% for Task 2, and an average of 4.76% for Task 3. This is because our GA-DTCDR effectively leverages the richness and diversity of the information on both domains, and intelligently and effectively combines the embeddings of common users.

Result 2: Ablation Study (for Q2)

To answer Q2, we implement a variant of our GA-DTCDR, i.e., GA-DTCDR Average, by replacing element-wise attention with average-pooling. Average-pooling is the combination strategy used by the existing dual-target CDR approaches [Zhu et al., 2019], which gives the weight equally, i.e., 0.5, to the embeddings of common users learned from dual domains. As we can see from Tables 4 - 6, with the element-wise attention, our GA-DTCDR improves GA-DTCDR Average by an average of 6.76%. This means that element-wise attention plays a very important role in our GA-DTCDR and the existing fixed combination strategies can hardly achieve an effective embedding optimization on each target domain.

Result 3: Impact of Embedding Dimension k (for Q3)

To answer Q3, we analyze the effect of k on the performance of our GA-DTCDR as depicted in Tables 4 - 6. In general, in terms of HR@10 and NDCG@10, the recommendation accuracy of our GA-DTCDR increases with k because a larger Task Domain SDR Baselines Single-Target CDR Baselines NeuMF

DMF

CTR-RBF BPR DCDCSR TMH

HR NDCG HR NDCG HR NDCG HR NDCG HR NDCG

Task 3

(k = 8)

DoubanMovie (Sparser) .5266 .2911 .5498 .3114 .5514 .3156 .5762 .3347 .5987 .3487

MovieLens (Richer)

.7818 .5024 .8115 .5219

-

-

-

-

-

-

Task 3

(k = 16)

DoubanMovie (Sparser) .5282 .2939 .5573 .3141 .5631 .3213 .5816 .3438 .6031 .3580

MovieLens (Richer)

.7901 .5084 .8143 .5212

-

-

-

-

-

-

Task 3

(k = 32)

DoubanMovie (Sparser) .5411 .2991 .5612 .3254 .5721 .3347 .5821 .3447 .6108 .3733*

MovieLens (Richer)

.7978 .5124 .8180 .5231*

-

-

-

-

-

-

Task 3

(k = 64)

DoubanMovie (Sparser) .5449 .3152 .5632 .3387 .5704 .3327 .5926 .3559 .6186 .3754*

MovieLens (Richer)

.7935 .5149 .8231* .5277

-

-

-

-

-

-

Task 3

(k = 128)

DoubanMovie (Sparser) .5512 .3301 .5776 .3505 .5912 .3741 .6142 .3904 .6314 .3927*

MovieLens (Richer)

.8042 .5205 .8319* .5344

-

-

-

-

-

-

embedding can represent a user/item more accurately. However, considering the structure of the neural network layers in Parameter Setting, the training time of our GA-DTCDR also increases with k. This is a trade-off. Therefore, considering both aspects, k = 64 is ideal in our experiments.

Result 4: Top-N Recommendation Performance (for Q4)

To answer Q4, we compare the performance of top-N recommendation in terms of HR@N and NDCG@N where N ranges from 1 to 10. In fact, the performance trends of the all top-N experiments (for all the tasks with different k) are similar. So, due to space limitation, we only report the TopN recommendation results of all the seven baseline models, GA-DTCDR Average, and GA-DTCDR for Task 1 (k = 8) in Fig. 2. In Fig. 2, on both DoubanBook (sparser) and DoubanMovie (richer), the performance of our GA-DTCDR is consistently better than those of all the seven baselines and GA-DTCDR Average. On DoubanBook, considering all the Top-N recommendations, our GA-DTCDR improves the best-performing baselines in different experimental cases by an average of 1.74% for HR@N, and by an average of 5.83% for NDCG@N, while on DoubanMovie, our GA-DTCDR

improves the best-performing baselines in different experimental cases by an average of 8.13% for HR@N, and by an average of 7.55% for NDCG@N.

5 Conclusion and FutureWork

In this paper, we have proposed a Graphical and Attention framework for Dual-Target Cross-Domain Recommendation, called GA-DTCDR, which proposes the graph embedding technique to generate more representative user and item embeddings and the element-wise attention mechanism to improve the recommendation accuracies on both domains simultaneously. Also, we have conducted extensive experiments to demonstrate the superior performance of our GADTCDR. In the future, we plan to extend our approach to multi-target recommendations and conduct more comprehensive experiments on new datasets to validate the impact of data sparsity and scale of common users on performance.

Acknowledgements

This work was supported by Australian Research Council Discovery Project DP180102378.