DeepAPF: Deep Attentive Probabilistic Factorization for Multi-site Video Recommendation

1 Introduction

In recent years, watching online videos has become increasingly popular in our daily activities [Ding et al., 2018]. This creates a large ecosystem with different content providers (CPs). In this ecosystem, CPs offer a variety of video contents for users. Since users freely watch their liked videos from various video sites, the competition for users is critically fierce among CPs. To achieve success, each CP takes efforts to provide the best video service. One of the most important strategies is designing smart personalized recommender systems to help users explore the videos of interest.

Most of recommender systems use one-site viewing data to predict user preferences on videos [Zhou et al., 2010; Qian et al., 2014; Bu et al., 2016]. In particular, some recent works apply Deep Neural Networks (DNNs) into the recommendation methods [Zhang et al., 2016; Zheng et al., 2017;

Zhang et al., 2017; Chen et al., 2017; Wang et al., 2017; Zhu et al., 2018; Zhang et al., 2018; Zhou et al., 2018; Gao et al., 2019; Fang et al., 2019]. For example, Covington et al. [Covington et al., 2016] propose a deep neural network for YouTube video recommendation. Song et al. [Song et al., 2016] model user temporal behavior by recurrent neural network (RNN) in recommender systems. Similarly, authors in [Gao et al., 2017] design a dynamic RNN to capture user temporal preferences in the video recommendation. Although most of them take efforts to improve recommendation performance, two important factors are neglected. First, many users visit multiple video sites to watch their favorite videos. Thus, it is insufficient to model user viewing behaviors in a single site. Second, some videos are distributed among multiple sites. It is common that one site may push videos that users have viewed in another site. By collaborating with the Internet Service Provider (ISP) in China, we obtain over 205 million user-video viewing records from six popular video sites. This provides us an opportunity to explore how user preferences distribute over multiple sites, and how multi-site data can be beneficial to the performance improvement of video recommendation. Yang et al. [Yang et al., 2017] give a preliminary analysis on multi-site user viewing behaviors. However, it only linearly learns the user-video interactions by inner product used in Matrix Factorization (MF) model, which cannot capture the complex interactions [He et al., 2017].

In this paper, we aim to accurately capture user preferences as well as model the complex user-video interactions based on the multi-site viewing records, which is very challenging. First, since MF has its limitation to model the complicated user-video interaction, how can we properly learn them? Second, users are free to watch videos in different sites, thus what are the features of user interests over multiple sites? How can we model them? Third, the importances of different features of user interests may be not equal with each other, and thus how to accurately learn them is the final challenge.

Inspired by [He et al., 2017], we use deep learning method to model the complex user-video interaction. Based on our analysis, we find that the cross-site commonality and site peculiarity are two main features of user interests. Specifically, in our dataset, users who have visited multiple sites (named multi-homed users) do not have consistent or independent user interests across sites. Videos that appear in multiple sites (named multi-homed vides) imply the consistence of user interests; while exclusive videos in each site contribute to the disparity of user interests. To learn the importances of these two features, we introduce the attention mechanism that is initially proposed to solve static control problem in neural machine translation [Bahdanau et al., 2014]. Finally, we propose a model of Deep Attentive Probabilistic Factorization (DeepAPF). It has ability to accurately capture user interests by learning the importances between cross-site commonality and site peculiarity, and approximate the complex interaction between users and videos to improve the recommendation performance.

Moreover, we conduct extensive experiments to evaluate the performance of our proposed model by comparing with different baseline models with that of our model based on the dataset from all six sites as well as from any pairs of two sites. The results show that DeepAPF achieves the best performance of the recommendation. Meanwhile, we also evaluate the impact of using attention mechanism. Compared with DeepAPF without attention, it can further improve the performance. Since using multi-site data is helpful for improving the recommendation performance, it can create the win-win chances for CPs to share viewing data via the trusted third party.

We summarize our main contributions as the following three aspects:

• We analyze the features of user preferences on videos over multiple sites. From the observations, we find that user-video interaction is complex. For multi-homed users, user preferences contain both common part and site-specific part with different importance for different users.
• We design a model of DeepAPF to accurately capture user preferences (including cross-site commonality and site peculiarity) as well as learn the complex user-video interactions in multiple sites.
• We conduct extensive experiments to evaluate the performance of our proposed DeepAPF. The results show that it achieves the best performance with the comparison of baselines.

2 Data Collection andMotivation

2.1 Dataset

We obtain the video viewing dataset via the collaboration with a major Internet Service Provider (ISP) in China. The dataset is collected at gateways deployed in the fixed networks of a large metropolis in China. Then, it is parsed by deep package inspection (DPI) appliances that have ability to parse the application layer protocol of data packets like Hyper Text Transfer Protocol (HTTP). Considering the protection of user privacy, ISP has anonymized the user information in the viewing logs before handing to us.

Overall, our dataset contains over 8 million users, 7.5 million videos and 205 million viewing logs between Nov. 1 and Dec. 31, 2014. Each log is comprised of the anonymized user identity (ID), access time and request Uniform Resource Locator (URL). By crawling the video URLs, we collect the basic information of viewed videos including the title, the type and the website where a video is requested. Since the access to large international video websites like YouTube is blocked by Great Firewall in China, we focus on 6 major domestic content providers (CPs) including Youku (YK), IQiyi (IQI), Sohu (SH), Kankan (KK), LeTV (LE), and Tencent Video (TC).

In addition, the videos are classified into 6 major types, which includes TV series, movie, news, cartoon, user generated content (UGC) and show. This classification is based on the video types labeled by each site. To distinguish the multihomed videos, we match their titles according to the specific naming rules. Specifically, based on our observation, we find that the CP's name is usually located at the beginning of the video titles. With this rule, we preprocess the video titles, and then check whether any two titles are the same. In this way, we accurately and effectively identify the multi-home videos.

Table 1 shows the number of users, videos and views in the 6 sites from our dataset. We observe that YK attracts most users to watch over 1 million videos; while KK has lowest user population in video viewing. This shows the differences exists between CPs.

Since users are free to watch videos in any sites, we can classify the users into two types of exclusive and multihomed. For an exclusive user, she/he only visits one site for video viewing. When a user visits more than one site, he/she is multi-homed. Similarly, the same videos may be uploaded in multiple sites, thus we would identify the exclusive and multi-homed videos. Overall, we obtain about 5 million multi-homed users (over half of total users), which indicates that it is common that users visit more than one site for video viewing. For multi-homed videos, although its number is small (3% of total videos) but they can draw 25% of views.

2.2 Motivation

Our goal is to effectively use multiple-site data to improve the performance of video recommendation, which has three challenges needed to address.

Challenge 1: Characterizing the complicated interaction between users and videos

Matrix Factorization (MF) has become popular in many practical recommender systems [Hu et al., 2018; He et al., 2016]. It maps users and videos into a common latent space, where a latent feature vector can be used to represent a user or a video. Then, the interaction ̂ sij between user i and video j can be formulated as

1

where ui and vj represent the L-dimensional latent feature vector of user i and video j, respectively. It can be seen that MF linearly model such interaction by calculating inner product of associated latent vectors. However, as analyzed in [He et al., 2017], it cannot sufficiently explore the complicated interactions between users and videos. Thus, how to characterize the complex interactions is a challenging problem.

Challenge 2: Finding the features of user interests across multiple sites

Most recommender systems use one-site viewing data to make video recommendation, which lacks the global information of user interests across different sites. One way to resolve this problem is to merge the multiple-site viewing data, which models user interests as one set of latent feature vectors that are site-agnostic. However, it is known that different sites have their own unique features. For example, TC, as a large social media company, actively pushes news videos at the news portal though the social network. To explore whether such differences exist, we perform the following empirical analysis.

We define a feature vector pu (i) to represent user interests of user u in site i, where each element is the number of views from a specified video type in this site. Since the videos are classified into six types in our dataset, the dimension of the feature vector is set to 6. For multi-homed users, we can obtain their feature vectors in different sites. Next, we compute Pearson product-moment correlation between pu (i) and pu (j) of multi-homed users.

In our experiment, we consider the multi-homed users who have more than 100 views, because inactive users are insufficient to reflect user interests. For a multi-homed user u, we average his/her correlation coefficients of different twosite pairs, and plot the results in Figure 1. We find that most of multi-homed users do not have consistent or independent preferences across sites. Further, we calculate the average site-site correlation, and list the results in Table 2. We observe that in each site-site pair, the correlation is neither 0 or 1, which indicates that user interests have the features of the cross-site commonality and site peculiarity. On one hand, different CPs have their specialized features (i.e., P2P downloading, social network construction) in popularizing their video services. On the other hand, multi-homed videos make it possible to share user interests across sites. Thus, the cross-site commonality and site peculiarity of user interests co-exist, and they are critically important for the design of our model. However, it is challenging to model them.

Challenge 3: Estimating the feature importances of user interests

As discussed above, the features of user interests contain two parts: the cross-site commonality and site peculiarity. Intuitively, they contribute differently for user interests. For example, as TC that pushes the news through social network, when predicting a user's interest on a news video in this site, it is reasonable that the importance of site peculiarity should be higher than that of cross-site commonality. Thus, the importances between the cross-site commonality and site peculiarity are non-uniform for different users, and needs to be dynamically adjusted. However, how to accurately learn the importance weights is another challenging problem.

3 Method

To solve the three challenges discussed in Section 2, we propose a model of DeepAPF for video recommender systems using the multi-site data. Figure 2 illustrates its architecture which includes the following three layers.

• Embedding layer. The sparse representations of users and videos are projected to the dense vectors. The user embedding is decomposed into two parts: common part and site-specific part, which addresses the second challenge.
• Attentive layer. To overcome the third challenge, we make use of an neural attentional network to estimate the varying importances of the two features of user interest across sites.

• Predict layer. We use a generalized matrix factorization (GMF) model [He et al., 2017] to learn the complex user-video interaction, which solves the first challenge.

3.1 Embedding Layer

First, we convert the identity of a user (i) and a video (j) and a site (k) to a sparse vector using one-hot encoding, which are formulated as follows,

where function one - hot(i) can generate a vector of all zero values except i-th element with value 1. U, V and K are the sets of users, videos and sites, respectively.

Then, the sparse vectors are projected to the dense ones. Since the cross-site commonality and site peculiarity co-exist for multi-homed users, the embedding vector uki of user i at site k is decomposed into the common part ¯ui ∈ RL and the site-specific part ∆uki ∈ RL, which are computed as follows,

where P ∈ R|U|×L and Xk ∈ R|U|×L are learnable parameters. We denote the total number of users and sites as |U| and |K|, respectively. ∆ui ∈ R|U|×|K| is combined by |K| site-specific embeddings of user i.

Meanwhile, we use vj to denote the L-dimensional video latent feature vector, which is computed as follows,

where Q ∈ R|V|×L are parameters to be learned. |V| is the total number of videos.

3.2 Attentive Layer

Since the cross-site commonality and site peculiarity differently contribute to user preference, we use a neural attention network that is widely used in natural language processing [Bahdanau et al., 2014] to estimate their influences. Specifically, we use the attention parameter αk i,m(m = 1, 2) to represent the importance of the common and site-specific part. This attention network uses ¯ui, ∆uki and vj as input, and exploits a multi-layer perceptron (MLP) to estimate the attention function f, which is formulated as

where ⊙ is the element-wise product of vectors. W denotes the weight matrix in the attention network. b and h represents the bias vector and a weight vector, respectively. For simplicity, we use ui to denote ¯ui or ∆uki . ReLU(·) is used as the activation function of Rectifier Linear Unit (ReLU) [Cao et al., 2018] for the hidden layer in the attention network. Inspired by [He et al., 2018], to avoid the large variance on attention weights for users, we smooth the softmax function used in (6), which is as follows

where β is the smoothing parameter, which ranges from 0 to 1. Especially, β = 1 represents the standard softmax function, and β < 1 would alleviate the punishment on the attention weights of active users. According to Equation (6) and (8), we obtain the user embedding at site k

3.3 Prediction Layer

At prediction layer, we use the user embedding uki and video embedding vj as input, and define a mapping function as:

Then, we estimate the prediction score ̂ yk i,j , which represents how likely user i prefers video j. We formulate it as:

where σ(·) and h′T denote the sigmoid function and the weight vector of the prediction layer, respectively.

3.4 Training

To learn the model parameters, we employ the log loss, an objective function, to minimize the distance between predicted score and target one, which is defined as:

where Z denotes the training set that contains the positive and negative samples generated from the viewing records. The log loss of Equation (12) is optimized by mini-batch Adam optimizer [Kingma and Ba, 2014] to learn the model parameters. To construct a mini-batch, we sample the positive samples from the viewing records, and negative samples from unobserved videos for the corresponding users. With the constructed mini-batch, we perform gradient descent approach to minimize the objective function.

3.5 Discussion

MPF is demonstrated to achieve better performance with comparison of existing popular approaches such as Merged Matrix Factorization (MMF) [Yang et al., 2017]. Now we prove how MPF can be regarded as a special case. According to the one-hot encoding of user or video identity, we can obtain the embedding vector that can be regarded as the latent feature vector. At the attentive layer, we enforce αk 1 be 0.5, thus uki = ¯ui +∆uki . Then, in Equation (11), we redefine σ(·) as the identify function, and use a full-one vector for h′ . Under these steps, the MPF model can be exactly recovered.

4 Experimental Design and Results

We now conduct several experiments to compare the recommendation performance of our DeepAPF model with the state-of-the-art recommendation models.

4.1 Experimental Design and Setting

In the experiments, we mainly focus on multi-homed users to evaluate the DeepAPF model, and videos with less than 100 views are removed. As our dataset only contains implicit feedback of video viewing, we use the top-N recommendation to measure the performance of our proposed model. Specifically, in the test set, the score ̂ ys ij that user i prefers video j in site s can be predicted. We sort ̂ ys ij by the descending order to obtain the top-N recommendation list, and choose the first N videos to recommend.

Then we describe how to generate the training set and test set. In each set, there are both positive samples and negative samples. We treat the viewing records as the positive samples, and users not viewing the videos as the negative samples. In the training set, the negative samples are 1.5 times as many as the positive samples per user. In the test set, the ratio of the negative samples over the positive samples per user is set to 10. To make the prediction rational, users and videos in the test set also appear in the training set.

Our methods are implemented based on Tensorflow. We use the Gaussian distribution with a mean of 0 and a standard deviation of 0.01 to initialize the embedding and hidden parameters. We set the dimension size L of the latent feature vectors to 64, and set the batch size to 256. The learning rate is initially set to 0.01. We tune the hyper-parameter β in the attention network to achieve the near optimal performance. Note that in this work, we omit testing different dimension sizes since the previous work [He et al., 2017] has discussed it.

Baselines. We compare our proposed DeepAPF model with the following methods:

• GMF. [He et al., 2017] This approach has ability to learn the complex user-item interaction of each site based on DNNs.
• MPF. [Yang et al., 2017] This is the state-of-the-art matrix factorization approach that captures both cross-site commonality and site peculiarity over multi-site data.
• DeepMMF. This approach merges the user viewing records from all sites, and uses the GMF method to learn one set of the site-agnostic user latent features and video latent features.

Evaluation Metric. Following [Yang et al., 2017], we adopt F-measure as the evaluation metric in our experiments. Thus, F-measure value can be formulated as

where U test denotes the user set in the test set. V test i and V rec i are the set of positive test samples and the set of top N recommended videos for user i, respectively. Because we draw N as the same as the number of positive test sample per user, the F-measure value is equal to the precision value and recall value. Therefore, it is effective to evaluate the model performance. As future work, we plan to use more metrics (e.g., Normalized Discounted Cumulative Gain (NDCG)) in our experiments.

4.2 Experimental Results

We first measure how the overall recommendation performance of DeepAPF perform with the comparison of the baselines based on the multi-site data.

Table 3 shows the performance of our DeepAPF model and other three baselines for each site based on the data from other 5 sites. We observe that the average F-measure of our DeepAPF model is 17.62%, 7.9% and 8.1% higher than MPF, DeepMMF and GMF, respectively. The results show that our model can more accurately capture the user preferences over multi-site data, and thus achieve the significant improvements. In addition, although DeepMMF merges the data from other sites, it does not always performwell for each site when compared with GMF. This shows that blindly merging the data cannot effectively achieve the performance gain. In contrast, it enlarges the approximate error due to the restriction of the model space of learning various patterns of each site.

Site-site Performance

To investigate how one site's recommendation performs with the help of the data from another site, we conduct the experiments with different pairs of sites.

First, we compute the improved rate of F-measure by DeepAPF over MPF, and show the results in Table 4. We observe that DeepAPF outperforms the MPF methods for all the site pairs. For example, the F-measure in KK improves 10.82% when collaborating with the data from IQI. This is because our proposed methods can learn the complex uservideo interactions. Further, the improved rate exhibits the differences among different pairs of sites. This indicates that the features (e.g., site peculiarity) in different sites have impact on the information transferring.

Next, we compare the F-measure achieved by DeepAPF with that by GMF, as shown in Table 5. It can be observed that DeepAPF achieves the improvement of F-measure over GMF, which indicates that using another site's data is beneficial for predicting user interests.

Finally, we make a comparison between the F-measure of DeepAPF and that of DeepMMF for each site by virtue of another site's data. From Table 6, we can see that the F-measure of DeepAPF is higher than that of DeepMMF for each site. This is because that DeepAPF simultaneously captures both the common part and site-specific part of user preferences.

The Impact of Attention Mechanism

To explore the impact of attention mechanism in video recommendation, we conduct two experiments to compare the F-measure of DeepAPF with attention with that of DeepAPF without attention.

Table 7 shows improved rate of F-measure by DeepAPF with attention over DeepAPF without attention with the help of another site's data. From the results, we observe that for some site pairs (e.g., SH-LE pair) using attention mechanism achieves a limited improvement of F-measure; while for KK, our model with attention has a great improvement by virtue of another site's data. This indicates that KK that attracts least views is highly sensitive to the importance between the sitepeculiarity and cross-site commonality.

Figure 3 depicts the F-measure of DeepAPF with attention and without attention mechanism for each site by virtue of the data from other 5 sites. From the results, we find that using attention mechanism can improve the performance of recommendation.

Summary and Implications

Based on the above experiments, we observe 1) the Fmeasure of DeepAPF outperforms other three factorization models by 17.62%, 7.9% and 8.1%; 2) by virtue of only onesite data, our proposed model achieves the best performance; 3) the DeepAPF with attention outperforms that without attention. These give us a valuable and insightful guideline that integrating the data from multiple sites is win-win for CPs. However, due to the data privacy of different CPs, it is difficult to directly share the one CP's data with another. A potential solution is to introduce a trusted third party in charge of data collection as well as model training. When the trained model is ready, the third part distributes it to each site to perform the video recommendation.

5 Conclusion

In this paper, we propose a model of DeepAPF that not only accurately captures both the cross-site and site-specific user interests, but also learns the complex user-video interactions. Extensive experimental results demonstrate that our model outperforms three state-of-art baseline methods. In future, we will focus on the privacy issue of data sharing between sites. We plan to study how to improve our model without directly using other sites' data, which can better achieve win-win situation for different sites.

Acknowledgements

This work was supported in part by the National Nature Science Foundation of China under 61861136003, 61621091 and 61673237, Beijing National Research Center for Information Science and Technology under 20031887521, and research fund of Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology.