PLASTIC: Prioritize Long and Short-term Information in Top-n Recommendation using Adversarial Training

1 Introduction

With the sheer volume of online information, much attention has been given to data-driven recommender systems. Numerous recommendation techniques have been developed. Three main categories of them are collaborative filtering methods, content-basedmethods and hybrid methods [Bobadilla et al., 2013; Lu et al., 2015]. In this paper, we aim to develop a method producing a ranked list of n items to a user by collaborative filtering that exploits historical user-item interactions.

Matrix factorization (MF) [Koren et al., 2009] is one of the most successful techniques in the practice of recommendation due to its simplicity, attractive accuracy and scalability. It models the user preference matrix approximately as a product of two lower-rank latent feature matrices representing user profiles and item profiles respectively. Matrix factorization is robust to the data sparsity and cold-start issues in the recommender system.

⇤Min Yang is corresponding author (min.yang@siat.ac.cn)

1 Codes are publicly available at https://goo.gl/WUVK9a.

Despite the appeal of matrix factorization, this technique does not explicitly consider the temporal variability of data [Wu et al., 2017]. Firstly, the popularity of an itemmay change over time. For example, movie popularity booms or fades, which can be triggered by external events such as the appearance of an actor in a new movie. Secondly, users may change their interests and baseline ratings over time. For instance, a user who tended to rate an average movie as "4 stars", may now rate such a movie as "3 stars". Recently, recurrent neural network (RNN) [Hochreiter and Schmidhuber, 1997] has gained significant attention by considering such temporal dynamics for both users and items, and achieved high recommendation quality [Wu et al., 2016b; 2017].

More recent work [Wu et al., 2017] reveals that matrix factorization based and RNN based recommendation approaches are complementary to each other. Specifically, the matrix factorization recommendation approaches make item predictions based on users' long-term interests which change very slowly with respect to time. On the contrary, the RNN recommendation approaches predict which item will the user consume next, respecting the dynamics of users' behaviors and items' attributes in the short term. It therefore motivates us to devise a joint approach that takes advantage of both matrix factorization and RNN, exploiting both long-term and short-term associations among users and items.

In this paper, we propose a novel PLASTIC model, which Prioritizes Long And Short-Term Information in topn reCommendation using adversarial training. The PLASTIC model employs an adversarial framework to combine the MF and RNN based models for the top-n recommendation, taking the best of each to improve the final recommending performance. In the adversarial process, we simultaneously train two models: a generative model G and a discriminative model D. In particular, the generator G takes the user ui and time t as input, and predicts the recommendation list for user i at time t based on the historical user-item interactions. We implement the discriminator D via a Siamese Network that incorporates long-term and session-based ranking model in a pair-wise scenario. The two point-wise networks of Siamese Network share the same set of parameters. The generator G and the discriminator D are optimized with a minimax twoplayer game. The discriminator D tries to distinguish the real high-rated items in the training data from the recommendation list generated by the generator G, while the training procedure of generator G is to maximize the probability of D making a mistake. Thus, this adversarial process can eventually adjust G to generate plausible and high-quality recommendation list.

We summarize our main contributions as follows:

• ÷ To the best of our knowledge, we are the first to use GAN framework to leverage the MF and RNN approaches for top-n recommendation. This joint model adaptively adjusts how the contributions of the long-term and shortterm information of users and items are mixed together.
• ÷ We propose four hard and soft mixture mechanisms to integrate MF and RNN. We use the hard mechanism to calculate the mixing score straightforwardly and explore several soft mechanisms to learn the temporal dynamics with the help of the long-term profiles.
• ÷ Our model uses reinforcement learning to optimize the generator G for generating highly rewarded recommendation list. Thus, it effectively bypasses the nondifferentiable task metric issue by directly performing policy gradient update.
• ÷ The experimental results show that our model consistently outperforms the state-of-the-art methods.

2 RelatedWork

Recommender system is an active research field. The authors of [Bobadilla et al., 2013; Lu et al., 2015] describe most of the existing techniques for recommender systems.

Matrix factorization for recommendation Modeling the long-term interests of users, the matrix factorization method and its variants have grown to become dominant in the literature [Rennie and Srebro, 2005; Koren et al., 2009; Hernando et al., 2016; He et al., 2016; Tu et al., 2015]. In the standard matrix factorization, the recommendation task can be formulated as inferring missing values of a partially observed user-item matrix [Koren et al., 2009]. Srebro et al. [2005] suggested the Maximum Margin Matrix Factorization (MMMF), which used low-norm instead of low-rank factorizations. Mnih and Salakhutdinov [2008] presented the Probabilistic Matrix Factorization (PMF) model that characterized the user preference matrix as a product of two lower-rank user and item matrices. The PMF model was especially effective at making better predictions for users with few ratings. He et al. [2016] proposed a new MF method which considers the implicit feedback for on-line recommendation.

Recurrent neural network for recommendation These traditional MF methods for recommendation systems are based on the assumption that the user interests and item attributes are near static, which is however not consistent with reality. Koren [2010] discussed the effect of temporal dynamics in recommender systems and proposed a temporal extension of the SVD++ (called TimeSVD++) to explicitly model the temporal bias in data. However, the features used in TimeSVD++ were hand-crafted and computationally expensive to obtain. Recently, there have been increasing interests in employing recurrent neural network to model temporal dynamic in recommendation systems. For example, Hidasi et al. [2015] applied recurrent neural network (GRU) to session-based recommender systems. Wu et al. [2016a] proposed a recurrent neural network to perform the time heterogeneous feedback recommendation. Wu et al. [2017] used LSTM autoregressive model for the user and item dynamics and employed matrix factorization to model the stationary components that encode fixed properties. Different from their work, we use GAN framework to leverage the MF and RNN approaches for top-n recommendation, aiming to generate plausible and high-quality recommendation lists.

Generative adversarial network for recommendation In parallel, previous work has demonstrated the effectiveness of generative adversarial network (GAN) [Goodfellow et al., 2014] in various tasks such as image generation [Reed et al., 2016], image captioning [Chen et al., 2017], and sequence generation[Yu et al., 2017; Liu et al., 2018]. The most related work to ours is [Wang et al., 2017], which proposed a novel IRGAN mechanism to iteratively optimize a generative retrieval component and a discriminative retrieval component. Our approach differs from theirs in several aspects. First, we combine the MF approach and the RNN approach with GAN, exploiting the performance contributions of both approaches. Second, IRGAN does not attempt to estimate the future behavior. In fact, they use future trajectories to infer the historical records, which seems not useful in real-life applications.

3 Our Model

Suppose there is a sparse user-item rating matrix R that consists of U users and M items. Each entry rij↪t denotes the rating of user i on item j at time step t. The rating is represented by numerical values from 1 to 5, where the higher value indicates the stronger preference. Instead of predicting the rating of a specific user-item pair as is done in [McNee et al., 2006], our model aims to provide users with ranked lists of items (top-n recommendation) [Liu and Yang, 2008].

The PLASTIC model employs an adversarial framework to combine the MF and RNN based models for the top-n item recommendation. In the adversarial process, we simultaneously train two models: a generative model G and a discriminative model D.

3.1 Matrix Factorization (MF)

The MF framework [Mnih and Salakhutdinov, 2008] models the long-term states (global information) for both users (eu ) and items (em). In its standard setting, the recommendation task can be formulated as inferring missing values of a partially observed user-item rating matrix R. The formulation of MF is given by:

where eui and emj represent the user and item latent factors in the shared d-dimension space respectively. rij denotes the user i's rating on item j. Iij is an indicator function and equals 1 if rij > 0, and 0 otherwise. λu and λm are regularization coefficients. The ⇢(·) is a logistic scoring function that bounds the range of outputs.

Because minimizing the objective function - the squared errors - does not perfectly align with the goal to optimize the ranking order. In this paper, we apply MF for ranking prediction (top-n recommendation) directly, similar to [Wang et al., 2017].

3.2 Recurrent Neural Network (RNN)

The RNN based recommender system focuses on modeling session-based trajectories instead of global (long-term) information [Wu et al., 2017]. It predicts future behaviors and provides users with a ranking list given the users' past history. The main purpose of using RNN is to capture timevarying state for both users and items. Particularly, we use LSTM cell as the basic RNN unit. Each LSTM unit at time t consists of a memory cell ct, an input gate it, a forget gate ft, and an output gate ot. These gates are computed from previous hidden state ht-1 and the current input xt:

The memory cell ct is updated by partially forgetting the existing memory and adding a new memory content lt:

Once the memory content of the LSTM unit is updated, the hidden state at time step t is given by:

For simplicity of notation, the update of the hidden states of LSTM at time step t is denoted as ht = LSTM(ht-1↪ xt).

Here, we use zui↪t 2 RM and zmj↪t 2 RU to represent the rating vector of user i and item j given time t respectively. Both zui↪t and zmj↪t serve as the input to the LSTM layer at time t to infer the new states of the user and the item:

Here, hui↪t and hmj↪t denote hidden states for user i and item j at time step t respectively.

3.3 RNN andMF Hybrid

The session-based model deals with temporal dynamics of the user and item states, we further incorporate the long-term preference of users and the fixed properties of items. To exploit their advantages together, similar to [Wu et al., 2017], we define the rating prediction function as:

where g(▷) is a score function, eui and emj denote the global latent factors of user i and item j learned by Eq. (1); hui↪t and hmj↪t denote the hidden states at time step t of two RNNs learned by Eq. (6) and Eq. (7) respectively. In this work, we study four strategies to calculate the score function g, integrating MF and RNN. The details are described below.

PLASTIC-V1 This is a hard mechanism, using a simple way to calculate the mixing score from MF and RNN with the following formulation:

where bui and bmj are the biases of user i and item j; hui↪t and hmj↪t are computed by Eq. (6) and Eq. (7).

In fact, PLASTIC-V1 does not exploit the global factors in learning the temporal dynamics. In this paper, we also design a soft mixture mechanism and provide three strategies to account for the global factors eui and emj in learning hui↪t and hmj↪t , as described below (i.e., PLASTIC-V2, PLASTICV3 and PLASTIC-V4).

PLASTIC-V2 We use the latent factors of user i (eui ) and item j (emj ) pre-trained by MF model to initialize the hidden states of the LSTM cells hui↪0 and hmj↪0 respectively.

PLASTIC-V3 We extend PLASTIC-V2 by treating eui (for user i) and emj (for item j) as the static context vectors, and feed them as an extra input into the computation of the temporal hidden states of users and items by LSTM. At each time step, the context information assists the inference of the hidden states of LSTM model.

PLASTIC-V4 This method uses an attention mechanism to compute a weight for each hidden state by exploiting the global factors. The mixing scores at time t can be reformulated by:

where cui↪t and cmj↪t are the context vectors at time step t for user i and item j; hui↪t and hmj↪t are the hidden states of LSTMs at time step t, computed by

The context vectors cui↪t and cmj↪t act as extra input in the computation of the hidden states in LSTMs to make sure that every time step of the LSTMs can get full information of the context (long-term information). The context vectors cui↪t and cmj↪t are the dynamic representations of the relevant long-term information for user i and item j at time t, calculated by

where U and M are the number of users and items. The attention weights ↵ik↪t and βjp↪t for user i and item j at time step t are computed by

where σ is a feed-forward neural network to produce a realvalued score. The attention weights ↵it and βj t together determine which user and item factors should be selected to generate rij↪t.

3.4 Generative Adversarial Network (GAN) for Recommendation

Generative adversarial network (GAN) [Goodfellow et al., 2014] consists of a generator G and a discriminator D that compete in a minimax game with two players: The discriminator tries to distinguish real high-rated items on training data from ranking or recommendation list predicted by G, and the generator tries to fool the discriminator to generate(predict) well-ranked recommendation list. Concretely, D and G play the following game on V(D,G):

Here, x is the input data from training set, z is the noise variable sampled from normal distribution.

We propose an adversarial framework to iteratively optimize two models: the generative model G predicting recommendation list given historical user-item interactions, and the discriminative model D predicting the relevance of the generated list. Like the standard generative adversarial networks (GANs) [Goodfellow et al., 2014], our model also optimizes the two models with a minimax two-player game. D tries to distinguish the real high-rated items in the training data from the recommendation list generated by G, while G maximizes the probability of D making a mistake. Hopefully, this adversarial process can eventually adjust G to generate plausible and high-quality recommendation list. We further elaborate the generator and discriminator below.

Discriminative Model

As depicted in Figure 1 (right side), we implement the discriminator D via a Siamese Network that incorporates long and session-based ranking models in a pair-wise scenario. The discriminator D has two symmetrical point-wise networks that share parameters and are updated by minimizing a pair-wise loss.

The objective of discriminator D is to maximize the probability of correctly distinguishing the ground truth items from generated recommendation items. For G fixed, we can obtain the optimal parameters for the discriminator D with the following formulation.

where U denotes the user set, ui denotes user i, m+ is a positive (high-rating) item, m- is a negative item randomly chosen from the entire negative (low-rating) item space, ✓ and φ are parameters of D and G, and mg↪t is the generated item by G given time t. Here, we adopt hinge loss as our training objective since it performs better than other training objectives. Hinge loss is widely adopted in various learning to rank scenario, which aims to penalize the examples that violate the margin constraint:

where ✏ is the hyper-parameter determining the margin of hinge loss, and we compress the outputs to the range of (0↪ 1).

Generative Model

Similar to conditional GANs proposed in [Mirza and Osindero, 2014], our generator G takes in the auxiliary information (user ui and time t) as input, and generates the ranking list for user i. Specifically, when D is optimized and fixed after computing Eq. 19, the generator G can be optimized by minimizing the following formulation:

Here, M denotes the item set. As in [Goodfellow et al., 2014], instead of minimizing log(1-D(ui↪ mg↪t↪ m+♣t)), we train G to maximize log(D(ui↪ mg↪t↪ m+♣t)).

Policy Gradient

Since the sampling of recommendation list by generator G is discrete, it cannot be directly optimized by gradient descent as in the standard GAN formulation. Therefore, we use policy gradient based reinforcement learning algorithm [Sutton et al., 2000] to optimize the generator G so as to generate highly rewarded recommendation list. Concretely, we have the following derivations:

where K is number of items sampled by the current version of generator and mk is the k-th sampled item. With reinforcement learning terminology, we treat the term logD(ui↪mk↪m+♣t) as the reward at time step t, and take an action mk at each time step. To accelerate the convergence, Algorithm 1: Long and Session-based Ranking Model with Adversarial Network

1 Input: generator Gφ, discriminator D✓, training data S. 2 Initialize models Gφ and D✓ with randomweights, and pre-train them on training data S. 3 repeat 4 for g-steps do 5 Generate recommendation list for user i at time t using the generator Gφ. 6 Sample K candidates from recommendation list. 7 for k 2 ¶1↪ ▷▷▷↪ K♦ do 8 Sample a positive item m+ from S. 9 Compute the reward logD(ui↪mk↪m+♣t) with Eq.(20) 10 Update generator Gφ via policy gradient Eq.(22). 11 for d-steps do 12 Use current Gφ to generate a negative item and combined with a positive item sampled from S. 13 Update discriminator D✓ with Eq.(19).

14 until convergence

the rewards within a batch are normalized with a Gaussian distribution to make the significant differences.

The overall procedure is summarized in Algorithm 1. During the training stage, the discriminator and the generator are trained alternatively in an adversarial manner via Eq.(19) and Eq.(22), respectively.

4 Experimental Setup

4.1 Datasets

In order to evaluate the effectiveness of our model, we conduct experiments on two widely-used real-life datasets: Movielens100K and Netflix (called "Netflix-Full"). For each dataset, we split the whole data into several training and testing intervals based on time, as done in [Wu et al., 2017], to simulate the actual situation of predicting future behaviors of users given the data that occurred strictly before current time step. Then, each testing interval is randomly divided into a validation set and a testing set. We removed the users and items that do not appear in training set from the validation and test sets. The detailed statistics are presented in Table ?? 2 . The users and items that are not shown in the training data are removed from the test data. Following [Wang et al., 2017], we treat "5-star" in Netflix, "4-start" and "5start" in Movielens100K as positive feedback and all others as unknown (negative) feedback.

4.2 Implementation Details

Matrix Factorization. We use matrix factorization with 5 and 16 factor numbers for Movielens and Netflix respectively [Wang et al., 2017]. The parameters are randomly initialized by a uniform distribution ranged in [-0.05, 0.05]. We take gradient-clipping to suppress the gradient to the range of [0.2,0.2]. L2 regularization (with λ = 0▷05) is used to the weights and biases of user and item factors.

2 "Density" shows the average amount of 5-ratings for the user per day. "Sparsity" shows the filling-rate of user-item rating matrix as used in [Wu et al., 2017]

Recurrent Neural Network. We use a single-layer LSTM with 10 hidden neurons, 15-dimensional input embeddings, and 4-dimensional dynamic states where each state contains 7-days users/items behavioral trajectories. That is, we take one month as the length of a session. The parameters are initialized in the same way as in MF. L2 regularization (with λ = 0▷05) is used to the weights and biases of the LSTM layer to avoid over-fitting.

Generative Adversarial Nets. We pre-train G and D on the training data with a pair-wise scenario, and use SGD algorithm with learning rate 1 ⇥ 10-4 to optimize its parameters. In addition, we use matrix factorization model to generate 25 candidate items, and then re-rank these items with LSTM. In all experiments, we conduct mini-batch training with batch size 128.

4.3 EvaluationMetrics

To quantitatively evaluate our method, we adopt the rankbased evaluation metrics to measure the performance of top-n recommendation [Liu and Yang, 2008], including Precision@N, Normalised Discounted Cumulative Gain (NDCG@N), Mean Average Precision (MAP) and Mean Reciprocal Ranking (MRR).

4.4 Comparison to Baselines

In the experiments, we compare our approach with several strong baseline methods including Bayesian Personalised Ranking (BPR) [Rendle et al., 2009], Pairwise Ranking Factorization Machine (PRFM) [Qiang et al., 2013], LambdaFM [Yuan et al., 2016], Recurrent Recommender Networks (RRN) [Wu et al., 2017], and IRGAN [Wang et al., 2017]. For all the baseline methods, we adopt the default settings for all hyperparameters.

5 Experimental Results

5.1 Quantitative Results

We first evaluate the performance of top-n recommendation. The experimental results are summarized in Tables ?? and 3. Our model substantially and consistently outperforms the baseline methods by a noticeable margin on all the experimental datasets. In particular, we have explored several versions of our model with different mixture mechanisms. As one anticipates, PLASTIC-V4 achieves the best results across all evaluation metrics and all datasets. For example, on MovieLens dataset, PLASTIC-V4 improves 7▷45% on percision@5 and 6▷87% on NDCG@5 over the baseline methods.

The main strength of our model comes from its capability of prioritizing both long-term and short-term information in top-n recommendation. In addition, our mixture mechanisms (hard and soft) also seem quite effective to integrate MF and RNN.

To better understand the adversarial training process, we visualize the learning curves of PLASTIC-V4 as shown in Figure 2. Due to the limited space, we only report the Pecision@5 and NDCG@5 scores as in [Wang et al., 2017]. The other metrics exhibit a similar trend. As shown in Figure 2, after about 50 epochs, both Precision@5 and NDCG@5 converge and the winning player is the generator which is used to generate recommendation list for our final top-n item recommendation. The performance of generator G becomes better with the effective feedback (reward) from discriminator D. On the other hand, once we have a set of high-quality recommendation movies, the performance of D deteriorates gradually in the training procedure and makes mistakes for predictions. In our experiments, we use the generator G with the best performance to predict test data.

5.2 Case study

In this section, we will further show the advantages of our models through some quintessential examples. In Table 4, we provide the recommendation lists generated by two strong competitors (i.e., IRGAN and RNN) as well as the proposed PLASTIC-V4 model for two users who are randomly selected from the Netflix-Full dataset. Our model can rank the positive movies in higher positions than compared methods. For example, some emerging movies such as "Session 9" and "The Last Valley" that are truly attractive to the first user have been recommended by our models, whereas they are ignored by the compared methods. In fact, we can include all positive movies in the top-10 list for the first user and in the top-3 list for the second user.

6 Conclusion

In this paper, we proposed a novel adversarial process for topn recommendation. Our models incorporate both matrix factorization and recurrent neural network to exploit the benefits of the long-term and short-term knowledge. Experiments on two real-life datasets showed the performance superiority of our models for top-n recommendation.