BPAM: Recommendation Based on BP Neural Network with Attention Mechanism

1 Introduction

The era of information explosion has arrived, people can hardly hit what they really prefer when dealing with a huge number of choices. To tackle this problem, personalized recommendation systems have been proposed and widely used in e-commerce platforms and news/music/movie/education platforms [Hu et al., 2019; Wang et al., 2018a; Huang et al., 2019]. Collaborative Filtering (CF) is one of most classical technologies in personalized recommendation systems, which infers users' preferences from historical behavior [He et al., 2018].

The traditional CF recommendation algorithms are generally divided into two categories: Matrix Factorization (MF) and neighborhood-based CF methods. The MF methods map the users and items into a common representation space. Then the users' ratings to items are modeled as the inner product of their latent vectors. However, the MF methods easily suffer from the sparsity issue due to the long-tailed distribution of rating data in the real-world applications [Hu et al., 2016]. On the other hand, the neighborhood-based CF algorithms predict the target ratings by averaging (weighted) ratings of similar entities (users or items). In [Wang et al., 2006], both user and item information are taken into account to improve the prediction quality, but it only achieves some performance improvement. This indicates that it is hard to predict the target ratings accurately by utilizing the linear combination of similar entities' ratings.

Inspired by the significant success of deep learning, some efforts have been made in utilizing the representation learning abilities of deep neural networks (DNNs) to learn users' preference [Xue et al., 2017; Deng et al., 2019; He et al., 2017]. In [Xue et al., 2017], a novel deep matrix factorization model with deep neural network was proposed, which mapped the users and items into a common low-dimensional space with non-linear projections to make prediction. However, the high computational and storage costs caused by the complex structures prevent it from applying to large data [Wang et al., 2018b]. Besides, there are only a relatively small number of ratings that can be fed into DNNs as the training samples due to the sparsity issue in recommendation systems, which can easily lead to over-fitting of DNNs with massive parameters.

This paper addresses the above issues by proposing a recommendation algorithm based on Back Propagation (BP) neural network [Goh, 1995] with Attention Mechanism [Wang et al., 2016] (BPAM). By introducing the BP neural network into the neighborhood-based CF algorithm, the ratings of similar users are fed into the BP neural network instead of undertaking a linear combination in the traditional algorithms. In this manner, BPAM is able to capture the nonlinear relationship between the target user and his/her neighbors. Considering the large number of items, we utilize the shallow network, i.e. BP neural network, to reduce the computational and storage costs. Unlike considering all ratings in the existing DNNs-based CF algorithms, the influence of non-similar users can be eliminated by selecting similar users' ratings. Moreover, since the BP neural network has a relatively small number of parameters, the over-fitting issues suffered by DNNs can be well avoided even in the case of only a relatively small number of ratings. In addition, the attention mechanism is incorporated into the BP neural network to capture the global impact of the target user's neighbors by means of introducing their global weights. In this way, a unified training model is constructed, which consists of the local weight and the global attention weight. When predicting the ratings of the target user, BPAM takes into account both local weight and global weight of the target user's neighbors to achieve global optimum.

The main contributions of this work are as follows.

• A novel neighborhood-based CF recommendation algorithm called BPAM is proposed, which overcomes the high computational and storage costs and over-fitting issues in DNNs.
• The BP neural network is utilized to learn the complex relationship between the target user and his/her neighbors instead of undertaking a linear combination in the traditional algorithms.
• A novel attention mechanism is introduced to capture the global impact of the target user's neighbors by means of introducing their global weights.
• Extensive experiments on eight real-world datasets are conducted to demonstrate the effectiveness of the proposed model. The results show that BPAM outperforms other state-of-the-art algorithms, and the proposed attention mechanism improves its performance significantly.

2 RelatedWork

Neighborhood-based CF algorithms are intuitive and interpretable, which calculate the similarity between entities (users or items) and then predict the ratings based on the similar entities. Various algorithms have been developed to improve the prediction accuracy from different aspects [Bell and Koren, 2007; Jia et al., 2010; Patra et al., 2015; Hu et al., 2019]. In [Bell and Koren, 2007], a method was proposed to simultaneously derive the interpolation weights as a global solution to an optimization problem, leading to some improvement of the prediction accuracy. In [Jia et al., 2010], the temporal information was utilized to improve the accuracy of CF algorithms. In [Patra et al., 2015], a similarity measure was proposed for neighborhood-based CF, which utilized all ratings information comprehensively for locating useful neighbors of an active user in the sparse rating matrix, and did not depend on co-rated items. However, these algorithms only achieve some performance improvement, since they still utilize the linear combination of similar entities' ratings to predict the target rating.

Recently, due to the representation learning abilities, DNNs have been introduced in recommender systems to learn users' preference for items [Xue et al., 2017; Wang et al., 2015; He et al., 2017; van den Oord et al., 2013]. DNNs are utilized to learn the complex mapping relationship between user-item latent factor representation and matching rating. For example, in [Xue et al., 2017], a deep learning architecture was presented to learn a common low dimensional space for the representations of users and items, where a two-pathway neural network architecture was used to replace the linear embedding operation. In [Wang et al., 2015], a Collaborative Deep Learning (CDL) was proposed, which jointly performed deep representation learning for the content information and collaborative filtering for the rating matrix. In [He et al., 2017], three instantiations, namely Generalized Matrix Factorization (GMF), Multi-Layer Perceptron (MLP) and Neural Matrix Factorization (NeuMF), were proposed, which modeled user-item interactions in different ways. These DNNs-based models have greatly improved the prediction accuracy, but they suffer from the issues of overfitting and high computational and storage costs.

Attention-based architectures, which learn to focus their "attention" to specific parts [Cheng et al., 2018] or combine both local and global information [Gong and Zhang, 2016], have shown great potential in recommendation algorithms. For instance, in [Cheng et al., 2018], the attention mechanism was introduced to capture the varying attention vectors of each specific user-item pair. In [Gong and Zhang, 2016], an effective attention-based Convolutional Neural Networks (CNNs) was proposed for performing the hashtag recommendation task, which combined the local attention channel and the global channel to obtain the final embedding of the microblog. Motivated by the successes of various models, we adopt the attention mechanism to combine the local weights and global attention weights of neighbors.

3 The ProposedModel

3.1 Preliminaries

Suppose that U and V are the user set and the item set respectively, following [Zhu et al., 2017], a user-item rating matrix R ∈ R|U|×|V| is constructed from users' explicit feedback as follows,

where ru,i represents the rating of user u to item i.

The task of recommendation algorithms is to estimate the missing ratings in the rating matrix R [Li et al., 2017]. The model-based methods generally assume that the models generate data in such a way as ˆ yu,i = f(u, i|Θ), where ˆ yu,i denotes the prediction of yu,i, i.e. the predicted rating of user u to item i, and f denotes the mapping function that maps the model input, e.g. the neighborhoods' ratings, to the predicted rating of the corresponding user-item pair by utilizing the model parameters Θ [Deng et al., 2019]. The mapping function of neighborhood-based CF is a linear combination of neighbors' ratings, where the model parameters are mainly the weights obtained from different similarity functions. These simple linear mapping functions are usually hard to make accurate rating prediction. On the other hand, the mapping functions of DNNs-based CF are DNNs, which are used to learn the complex mapping relationship between user-item latent factor representation and matching rating. However, due to the sparsity issue in recommendation system, there are only a relatively small number of ratings, which are fed into DNNs with large number of parameters, leading to overfitting. Moreover, the training process of DNNs usually results in high computational and storage costs. In this paper, we utilize the BP neural network as mapping function to learn the complex relationship of the target user and his/her neighbors.The BP neural network has shown the ability of highly non-linear mapping, and can well fit the non-linear relationship between the target users and their neighbors [Goh, 1995]. Most importantly, it is a shallow network, which can be trained efficiently and make accurate prediction without a large number of samples.

3.2 Construction of KNNRatingMatrix

The general process for BPAM is illustrated in Figure 1. After extracting the rating matrix R from the database, we obtain the K-Nearest Neighbors (KNN) for each user by calculating the cosine similarity between users [Sarwar et al., 2001]. The KNN rating matrix for each user is formed by intercepting the rating information of the target user and his/her k nearest neighbors, where the KNN rating matrix for user u is denoted by N(u) ∈ R|V|×(k+1) . The last column in N(u) denotes the ratings of the target user u to items, which can be used as labels for training and testing, and the first k columns are the ratings of his/her neighbors to items. The i-th row N(u) i∗ ∈ N(u) corresponds to item i, which is regarded as a training sample (X(u) i∗ , y(u) i∗ ) of the BP neural network. y(u) i∗ and X(u) i∗ are the ratings of the target user u and his/her neighbors to item i respectively. In the local training process, the error between ˆy(u) i∗ and y(u) i∗ will be fed back to adjust the parameters.

3.3 Data Processing

Recommendation algorithms are known to suffer from serious sparsity problem. That is, the number of ratings per user obeys the long-tailed distribution [Jing et al., 2015]. Hence most of the values in the training sample vector are zero.

However, some users prefer to rate higher, while others prefer to rate lower. If feeding different users' zero-rating into the BP neural network, the preference biases won't be distinguished. To this end, we obtain the j-th value x′(u) i,j of the new input vector X′(u) i∗ by

where N(u) ∗j is the j-th column of N(u) , that is, the j-th nearest neighbor's ratings of user u. mean(N(u) ∗j ) represents the mean value of nonzero elements of N(u) ∗j . In this way, a new KNN rating matrix N′(u) is formed.

3.4 BP Neural Network with AttentionMechanism

By introducing the attention mechanism [Chen et al., 2017], the proposed BPAM model is able to consider not only the local weight of the neighbors to the target user, but also the global attention weight of these neighbors to all their nearest target users. For instance, assume that the nearest target user set of user u is T = {U1, U2, ..., Ut}, that is, user u is one of the k nearest neighbors of these target users. When predicting the ratings of the target user U1, we regard the impact of user u on U1 as the local weight, and the impact on T as the global weight. In this manner, we can prevent the model from falling into the local optimum. The architecture of the BPAMmodel is illustrated in Figure 2, where the input layer, hidden layer and output layer are the three layers in the BP neural network. A ∈ R|U|×q denotes the global attention weight matrix for all users, where q is the number of neurons in the hidden layer. The i-th row Ai∗ ∈ A denotes the global attention weight of the user i to his/her nearest target user set. W(u) ih ∈ Rk×q and W(u) ia ∈ Rk×q denote the local weight matrix and the global weight matrix of user u's neighbors respectively. The local weight and the global weight are combined to predict the ratings of the target user.

The training set D = {(X′(u) 1∗ , y(u) 1∗ ), ..., (X′(u) mu∗ , y(u) mu∗)} is composed of some rows in the new KNN rating matrix N′(u) obtained from Eq. (2). In these rows, the target user has rated the corresponding items and mu is the number of the target user's ratings. The training process of BPAM is divided into two stages, forward propagation and error back propagation [Li et al., 2012]. The forward propagation, i.e. the rating prediction process, can be defined as:

where b(u) ih denotes the bias vector among the input layer and hidden layer, W(u) ho and b(u) ho denote the weight matrix and the bias vector among the hidden layer and output layer. α is the trade-off parameter which is used to tune the importance of the global weight. W(u) ho , W(u) ih , b(u) ih and b(u) ho are initialized randomly before training. But W(u) ia is formed by intercepting the global attention weights of the user u's neighbors from A as follows,W(u)

where W(u) ia [t] denotes the t-th row ofW(u) ia and neighbor(t) denotes the row-index in A of the t-th nearest neighbor of the target user. The objective function of our BPAMmodel is defined as follows:

where

where the regularization terms are added to prevent overfitting. BPAM adjusts parameters in the direction of the negative gradient of the objective value based on Stochastic Gradient Descent (SGD). For training sample (X′(u) i∗ , y(u) i∗ ), the error back propagation, i.e. parameter update, can be defined as:

where η ∈ (0, 1) is the learning rate, and

After training the local BP neural network of user u, the global attention weight of neighbors in A will be updated as follows:

In summary, by introducing the attention mechanism, BPAM combines the local weight and the global weight to predict the missing ratings. The training process of BPAM is to alternately iterate the forward propagation and the error back propagation until the stopping condition is reached. The whole procedure of BPAM is summarized in Algorithm 1. Finally, the predicted ratings can be obtained via Eq. (3).

Algorithm 1 The algorithm framework of BPAM

Input: R: rating matrix; k: number of neighbors; α: tradeoff parameter.

Output: W: BP neural network weights; A: global attention weight; b: bias vector.

• 1: Randomly initializeW, A and b
• 2: Obtain the k-nearest neighbors for each user by calculating the cosine similarity between users
• 3: repeat
• 4: for all u ∈ U do
• 5: Form the KNN rating matrix N(u)
• 6: Form the new KNN rating matrix N′(u) via Eq. (2)
• 7: Construct the training set D fromN′(u)
• 8: Construct the global weight W(u) ia from A via Eq. (4)
• 9: for all (X′(u) i∗ , y(u) i∗ ) ∈ D do
• 10: Predict the rating ˆy(u) i∗ via Eq. (3)
• 11: Update W(u) ih , b(u) ih ,W(u) ho , b(u) ho , W(u) ia via Eq. (7) and Eq. (8)
• 12: end for
• 13: UpdateAaccording to the updatedW(u) ia via Eq. (9) 14: end for
• 15: until Eq. (5) converges
• 16: ReturnW, A, b

4 Experiments

4.1 Experimental Setting

Dataset. The experiments are conducted on eight realworld publicly available datasets: MovieLens (ml-latest (mlla), ml-1m, ml-10m) 1 , filmtrust 2 , jester (jester-data-1 (jd-1), jester-data-2 (jd-2), jester-data-3 (jd-3)) 3 and MovieTweetings (MT) 4 . The statistics of these eight datasets are summarized in Table 1. Notice that in order to verify the accuracy and effectiveness of the proposed model in the case of different data sizes (measured by the number of users and items), experiments are carried out on both large-scale datasets, namely ml-10m and MT, and other small datasets, respectively. We randomly split each dataset into the training set and testing set with ratio 3:1 for each user.

1 https://grouplens.org/datasets/movielens/

2 https://www.librec.net/datasets.html

3 http://eigentaste.berkeley.edu/dataset/

4 https://github.com/sidooms/MovieTweetings

Evaluation measures. We utilize the root-mean-square error (RMSE) and mean-absolute-error (MAE) to evaluate the performance of the predicted results. Smaller values of RMSE and MAE indicate the better performance.

4.2 Comparison Results

We compare the proposed BPAM method with the following five methods:

• UserCF [Herlocker et al., 1999] is a typical recommendation algorithm, which predicts the ratings of the target users based on the ratings of similar users. It is usually utilized as a benchmark of recommendation systems.
• PMF [Mnih and Salakhutdinov, 2008] is a probabilistic

algorithm that adopts a probabilistic linear model with Gaussian observation noise to model the user preference matrix as a product of lower-rank user matrix and item matrix.

• DeepCF [Deng et al., 2019] incorporates collaborative filtering methods based on representation learning and matching function learning to learn the complex matching function and low-rank relations between users and items.
• NeuMF [He et al., 2017] combines Generalized Matrix Factorization (GMF) and Multi-Layer Perceptron (MLP) to learn the user-item interaction function.
• DMF [Xue et al., 2017] utilizes deep neural network to learn a common low dimensional space for the representations of users and items. It uses a two-pathway neural network architecture to replace the linear embedding operation used in vanilla matrix factorization.

The comparison results are shown in Table 2. Memory error occurs in DNNs-based models, namely DeepCF, NeuMF and DMF, when training large-scale data due to a mass of parameters. Therefore, we record their corresponding unpredictable results as "NA" in Table 2. Since different datasets have different rating scales, as shown in the "Scale" column in Table 1, the RMSE and MAE values have exhibited some numerical differences on different datasets.

According to Table 2, we have the following key observations. Overall, the proposed BPAM model has achieved significant improvements over the second best method (which may vary from one dataset to another) on most of the datasets in terms of both RMSE and MAE. The only exception occurs on ml-la and ml-1m in terms of RMSE, where only less than 6% least improvements have been achieved. When compared with the five methods, extremely good improvements have been achieved with at least 41.38% average improvements. Another observation is that on the MovieLens datasets, namely ml-la, ml-1m and ml-10m, as the data size increases (from ml-la to ml-10m), the proposed BPAM model still makes significant improvements. However, the three deep neural network-based algorithms, namely DeepCF, NeuMF and DMF, fail to generate predicted ratings on ml-10m due to memory error. Similarly, on the another large-scale dataset, namely MT, the three existing DNNs-based CF algorithms also fail to generate predicted ratings. In addition, even on the small datasets, the BPAM model still outperforms the three existing deep neural network-based algorithms. This result has confirmed the effectiveness of utilizing shallow BP neural network and attention mechanism in recommender systems.

4.3 Impact of AttentionMechanism and Data Processing

In the design of BPAM, we utilize data processing to represent users' rating preferences (higher or lower) and introduce the global attention weights to prevent the model from local optimum. To validate the effectiveness of data processing and attention mechanism in our model, we compare our model with the following three variants: BPCF-WP (without data processing and attention mechanism), BPCF (without attention mechanism), BPAM-WP (without data processing). As shown in Table 3, we can observe that: (1) The methods with data processing perform a little better than those without data processing, especially on the MT dataset. It demonstrates the effectiveness of data processing before the input vectors are fed into the BP neural network. (2) BPAM outperforms BPCF with a large margin in all cases, which validates that our attention mechanism in BPAM can effectively capture the global attention weights about users' impacts on their nearest target user set.

4.4 Sensitivity Analysis of Hyper-parameters

In this section, we analyze the impact of the two hyperparameters: k and α by using the heat map. According to the results in Figure 3, the proposed model generates the best performance with k = 5 on most of the datasets except ml-1m. On ml-1m, the best performance is achieved with k = 10. Smaller number of neighbors usually performs better since it leads to fewer parameters and a relatively small number of samples are sufficient for training. Additionally, we can find that the optimal attention ratio α is around 2 to 4. And compared with α = 0.5, the values of RMSE and MAE decrease significantly, which indicates the significance of the global attention weight in BPAM.

5 Conclusion

In this paper, we propose a novel recommendation algorithm based on BP neural network with Attention Mechanism (BPAM). In the proposed algorithm, the BP neural network is utilized to learn the complex relationship between the target users and their neighbors. Compared with DNNs, shallow BP neural network can not only reduce the computational and storage costs, but also prevent the model from over-fitting caused by the small number of ratings. Besides, an attention mechanism is introduced in BPAM to capture the global attention weights about users' impact on their nearest target user set. Extensive experiments on eight benchmark datasets demonstrate that our proposed model distinctly outperforms state-of-the-art methods.

Acknowledgements

This work was supported by NSFC (61876193), Guangdong Natural Science Funds for Distinguished Young Scholar (2016A030306014), Tip-top Scientific and Technical Innovative Youth Talents of Guangdong special support program (2016TQ03X542), and Key Areas Research and Development Program of Guangdong (2018B010109007).