Pattern-enhanced Contrastive Policy Learning Network for Sequential Recommendation

1 Introduction

Sequential recommendation, which aims to recommend the next few items that the user will likely interact with in the near future, now is playing an increasingly important role in various application scenarios [Zhang et al., 2019b]. For this task, sequential dependency has been proven to be an important factor since items interacted in future are largely dependent on the previous ones [Kang and McAuley, 2018]. However, due to the randomness of the user's behaviors and his/her diverse interests, not all historical items contain useful signals w.r.t. the preference estimation for the target item. When taking these irrelevant yet noisy items into consideration, it becomes hard to precisely distill preference information for better recommendation performance. A good recommender should be able to extract these credible sequential dependencies relevant with the target item. Here, in this paper, we refer to this process as a sequence denoising task. Although it is appealing in theory, limitation and complication of user-item interactions make the denoising process non-trivial.

∗Corresponding author.

Over the past decades, there have been many research efforts related to sequence denoising in one form or another. Earlier works, including association rule [Lin et al., 2002] and sequential pattern mining [Mooney and Roddick, 2013], can be casted as a particular denoising process by extracting plausible patterns. With the prosperity of deep neural network, RNN-based and CNN-based models have become two mainstreams to distill discriminative information for recommendation. These methods mostly have a deficiency of treating each item equally, which may introduce much noise into the model learning process. Later, attentionbased approaches are proposed [Kang and McAuley, 2018; Yu et al., 2019] to estimate an importance weight for each item for better recommendation. Though effective, the existence of noisy data may still complicate the preference learning process. In addition, the interpretability is also limited as they fail to explicitly extract the truly relevant patterns. How to automatically extract credible sequential dependencies relevant with the target item for a correct recommendation, is a crucial and challenging problem.

In this paper, we propose a PatteRn-enhanced ContrAstive Policy Learning Network (named RAP) for sequence denoising. Specifically, RAP consists of two modules: a patternenhanced policy module and a contrastive learning module. The policy module employs a stochastic policy to sample action for each item from the initial sequence to identify its relevance with the target item. It is worthwhile to note that we have no access to relevance information for a given historical sequence and target item pair. To better facilitate denoising, we further mine a series of sequential patterns, and feed this global prior knowledge into policy network for guidance. According to the sampled actions, RAP produces a positive subsequence and a negative subsequence respectively. It is expected that the positive subsequence containing only the relevant items in the chronological order would ease the preference learning. To achieve effective denoising and preference estimation, a contrastive learning module is then applied on the resultant two subsequences with delayed rewards to make a robust gradient estimation. Empirical results on four realworld datasets demonstrate that the process of sequence denoising is feasible and our proposed method can outperform many state-of-the-art baselines significantly. We provide detailed analysis on the proposed model, and conduct case studies to gain better understanding on the learned patterns.

2 RelatedWork

In this section, we briefly review three lines of related work, i.e., sequential recommendation, RL-based recommendation and contrastive learning.

2.1 Sequential Recommendation

Sequential recommendation aims to predict users' future behaviors given their historical interaction data. Some earlier sequential recommendation methods rely on item-item transition matrices to capture the sequential patterns in the user interaction sequence [Rendle et al., 2010; Cheng et al., 2013]. Due to the nonlinear expressive capacity, RNN-based and CNN-based models have been widely utilized to model the interaction sequences. Specifically, RNN-based models focus on exploiting sequential dependencies from interaction sequences for recommendation [Hidasi et al., 2016; Huang et al., 2018], while the CNN-based methods allow feature composition over the consecutive behaviors in interaction sequence [Tang and Wang, 2018].

Though effective, none of the above works have introspected the rationality of fusing all historical interactions into account for the preference estimation. Recently, the attention mechanism has shown promising potential to support a context-aware preference learning for recommendation [Wang et al., 2018; Kang and McAuley, 2018]. However, these approaches still fail to obtain the truly relevant items for further consideration, which limits the recommendation performance and interpretability.

2.2 RL-based Recommendation

Reinforcement Learning aims to find an optimal action in a particular situation that would eventually maximize the longterm outcome. It has been introduced into recommender systems and achieved great success as its advantage of considering users' long-term feedbacks [Zhao et al., 2018b; Zou et al., 2020]. For example, [Zhao et al., 2018a] propose a deep reinforcement learning model to automatically learn the optimal strategies for page-wise recommendation. [Xian et al., 2019] propose a policy-gradient approach to extract paths from Knowledge Graph(KG), and take these paths as the recommendation interpretation. [Zheng et al., 2018] design a DQN-based reinforcement learning framework to make online personalized news recommendation. [Zou et al., 2019] formulate the ranking process as a multi-agent Markov Decision Process, where mutual interactions between documents are incorporated. [Zhang et al., 2019a] propose a hierarchical reinforcement learning to revise user profile for course recommendation. [Xin et al., 2020] propose a self-supervised reinforcement learning framework for sequential recommendation. Recently, the contextual information is also considered to enhance the reinforcement learning process for recommendation. For instance, [Zhou et al., 2020] investigate the potential of leveraging KG to improve the sample efficiency of RL methods.

2.3 Contrastive Learning

Contrastive learning has recently achieved great success in various domains [Dai and Lin, 2017; Laskin et al., 2020]. The key idea of contrastive learning is to perform unsupervised learning over semantically similar (positive) and dissimilar (negative) instance pairs. By encouraging the representation of similar pairs to be close, and those of dissimilar pairs to be far apart [Chuang et al., 2020], contrastive learning can alleviate the extensive demand for human annotations, and enable mining the underlying correlations behind the rich structure inside data [Devlin et al., 2019]. Inspired by these advances, we present the first contrastive learning based sequence denoising model for sequential recommendation.

3 ProblemDefinition

Let U={u1, u2, · · · , u|U|} denote a set of users, E={e1, e2, · · · , e|E|} be a set of items, where |U| and |E| represent the total number of unique users and items, respectively. For each user u ∈ U, we use eu={eu1 , eu2 , · · · , eun } to denote his/her interaction sequence, where eui ∈ E is the item that u has interacted with at time step i and n is the length of the interaction sequence eu .

In this paper, we consider an episodic RL approach to address the sequence denoising for sequential recommendation. Given a target item et, at each time step i, the process is in some state si ∈ S. According to state si, the agent performs an action ai modeled by a policy π(ai|si). The action space is ai ∈ {positive = 1, negative = 0}, where we use positive action to indicate that the item is relevant with the target item, and a negative action indicates that the item should be removed from the initial sequence. Given the definition of states and actions, the probability of choosing ai can be written as follows:

where si is the representation of state si, w1 and b denote parameters of the policy module. After sequence denoising, we can obtain a positive subsequence e+ and a negative subsequence e- from the initial sequence eu . The subsequence e+ is a chronologically ranked list of all relevant items. In contrast, e- is the list of all irrelevant items. Our aim is to infer the likelihood that the user will interact with et based on subsequence e+.

4 Pattern-enhanced Contrastive Policy Learning Network

In this section, we introduce our Pattern-enhanced Contrastive Policy Learning Network (RAP) in detail. Figure 1

shows the architecture of RAP, which consists of two main modules: 1) a pattern-enhanced policy module to sample action for each item in the initial interaction sequence, and split the initial sequence into two subsequences, which are expected to keep relevant or irrelevant items respectively; 2) a contrastive learning module is further applied on these two subsequences to generate a contrastive loss to guide the model optimization. For simplicity, we describe algorithmic details for a single user u, and it is straightforward to extend the formulas to a set of users. For brevity, we will omit the superscript u in the notations.

4.1 Pattern-enhanced PolicyModule

RAP utilizes a policy module to determine historical items' relevance with the target item. However, running a complicated policy on a sequence with sparse user-item interactions is not easy. In addition, this process is not guaranteed since no item-level relevance information is provided to supervise the denoising process. Considering the collective wisdom (i.e., sequential patterns) is well expressed in massive user behaviors, we choose to mine sequential patterns, and treat them as prior knowledge to guide the denoising process.

Specifically, we first use SPADE [Zaki, 2001] to extract the eligible patterns from the interaction sequences. We compile the extracted k-length patterns into a matrix Ck, where cik represents the i-th row of Ck, and the j-th element of cik is the number of patterns that start from ei and end with ej. Given an item ei, we use Ci = [ci1 , ci2 , ..., cik ] to express the corresponding pattern information, and further feed it into the calculation of policy state si to enrich its semantic:

where GRU(·) is the Gated Recurrent Unit (GRU), MLP(·) represents a multi-layer perception, vei is the embedding vector of item ei, I(·) is an indicator function, and ⊕ is a concatenation operator.

After sampling ai with Equation (1), we can decide whether to keep the item for preference estimation. Specifically, when ai = 1, the item is relevant and retained. On the other hand, ai = 0 suggests the opposite. In the latter case, si is set to be the same as si-1. Note that when i = 0, the state is initialized as s0 = vet , where et is the target item. In this sense, we inject the target item et into policy component to enable a context-aware denoising process.

By feeding the pattern information into the state representation, our policy can foresee future sequential information when making a decision, which could offer insightful clues to determine item-level relevance without direct supervision signals.

4.2 Contrastive LearningModule

Given an interaction sequence e, we use π(ai|si) to sample an action for each constituent item sequentially. That is, we can obtain an action list (a1, · · · , an). In this way, we naturally separate the initial sequence e into two subsequences: positive subsequence e+ and negative subsequence e- as aforementioned. Specifically, given a sampled action list (a1, · · · , an), the probability of generating e+ can be calculated as follows:

Moreover, based on the series of sampled e+, the objective function can be written as follows:

where r+ is the delayed reward over e+. As e+ is expected to precisely capture informative semantics that are relevant with the target item et, a simple solution is to encode positive subsequence e+. Here, we choose another GRU to derive the subsequence representation v+ as the hidden state of the last item in e+. Then, we use cosine similarity between e+ and et as the reward:

However, this strategy seems not an optimal option as the optimization does not utilize irrelevant items in e, which also aggravates the sparse problem. In addition, due to the lack of supervision signals, a wrong distinction of our policy module (i.e., determining a relevant item as irrelevant) will also adversely affect performance. To fully utilize both relevant and irrelevant items, inspired by [Chen et al., 2018], we introduce a contrastive learning process to enhance model learning.

Here, we further adopt another GRU to encode the negative subsequence as v-. The corresponding probability of generating negative subsequence e- is as follows:

Similarly, r- is calculated as the cosine similarity between v- and vet . The final objective function is rewritten as follows:

4.3 Learning and Prediction

We use the Adam optimizer to maximize the objective function over all training instances. With the learned RAP model, given the interaction sequence of a user and the candidate items, we first scan the whole sequence according to Equation (2), and select each action with the maximal probability, which can be written as follows:

By this we extract the truly relevant items, and calculate its reward r+ according to Equation (3). We then rank all candidate items according to their rewards, and return the top-N as the final recommendations.

5 Experiments

In this section, we evaluate our proposed RAP 1 against several state-of-the-art methods. We first describe the datasets, baseline methods, evaluation metrics used in the experiments, and then analyze the experimental results.

5.1 Experimental Setup

Datasets

We conduct experiments on four publicly available real-world datasets from three different domains. Beauty and Games are two different categories of Amazon dataset 2 , which span from May 1996 to July 2014. LastFM 3 is a music listening dataset released from Last.fm online music system and we take a subset which contains interactions from Jan 2015 to June 2015. MovieLens is a popular benchmark dataset collected from the MovieLens website. In this work, we adopt ML-1m 4 which has one million user-movie interactions.

1 The implementation of our model is available at https://github. com/heilsvastika/RAP

2 http://jmcauley.ucsd.edu/data/amazon/

3 http://www.cp.jku.at/datasets/LFM-1b/

4 https://grouplens.org/datasets/movielens/1m/

Preprocessing

We follow the same preprossessing procedure in [Rendle et al., 2010; He and McAuley, 2016]. For all datasets, we use timestamps to order the interactions of each user. We discard users and items with fewer than 5 interactions. For partitioning, we split the historical sequence for each user u into three parts: (1) the most recent interaction for testing; (2) the nextto-last interactions for validation; and (3) all remaining interactions for training. This leave-one-out evaluation strategy has been widely used in related works [Kang and McAuley, 2018; Tang and Wang, 2018]. The detailed statistics for the four datasets are reported in Table 1.

Baseline Methods

We adopt the following representative and state-of-the-art methods as baselines for performance comparison, including non-sequential models and sequential models:

For non-sequential models, we consider BPR and NCF:

• BPR-MF: The Bayesian Personalized Ranking based Matrix Factorization [Rendle et al., 2009] is a conventional method for learning pairwise personalized rankings from user implicit feedback.
• NCF: The Neural Collaborative Filtering [He et al., 2017b] models user-item interactions with a series of nonlinear transformations.

For sequential models, we consider both shallow and deep models:

• FPMC: It captures users' general taste as well as their sequential behaviors by combining MF with first-order Markov Chains (MC) [Rendle et al., 2010].
• GRU4Rec: It uses GRU with ranking based loss to model user sequences for session-based recommendation [Hidasi et al., 2016].
• Caser: It employs CNN in both horizontal and vertical way to model high-order Markov Chains for sequential recommendation [Tang and Wang, 2018].
• NARM: It employs RNNs with attention mechanism to capture the users' preference and sequential behaviors [Li et al., 2017].
• SASRec: It uses a left-to-right Transformer model to capture users' sequential behaviors for recommendation [Kang and McAuley, 2018].
• HRL: It utilizes a hierarchical reinforcement learning algorithm [Zhang et al., 2019a] to revise the user profiles and tune the recommendation model on the revised profiles.

• SASRec+: We design an extension of SASRec by directly deleting the items with their attention scores smaller than a given threshold. Comparing with SASRec, in SASRec+, items with smaller attention scores have no influences towards preference learning.
• BERT4Rec: It employs the deep bidirectional selfattention to model user sequences and achieves state-ofthe-art recommendation performance [Sun et al., 2019].

For NCF, GRU4Rec, Caser, SASRec and BERT4Rec, we use the implementations released by the corresponding authors. FPMC and NARM are implemented by RecBole [Zhao et al., 2020]. For the rest, we implement them using PyTorch.

Evaluation Metrics

To conduct the performance evaluation, we adopt two widely used metrics: Hit Ratio(HR) and NDCG [He et al., 2017b; He et al., 2017a]. Considering we only have one test item for each user, HR@N is equivalent to Recall@N and proportional to Precision@N. In this work, we report HR and NDCG with N = 10. With paired t-test, performance differences are considered statistically significant with 0.05 level. We pair each ground-truth item in the test set with 1, 000 randomly negative items that the user has not interacted with, and rank these items with the ground-truth item together [Krichene and Rendle, 2020].

Implementation Details

We optimize all models with Adam optimizer [Kingma and Ba, 2015]. For baselines, we optimize each of them according to the validation sets. For SASRec+, the attention threshold is set to 0.015 also according to the validation sets. As to RAP, the hidden layer size of GRU and the embedding size are set to 100, the learning rate is set to 0.001, and the batch size is 128. The maximum length of sequential patterns extracted by SPADE is 5 (i.e., k = 5) when setting the support count to 2.

5.2 Performance Comparison

The overall performance in terms of HR@10 and NDCG@10 is shown in Table 2. Here, we make the following observations:

For the conventional models, we see that the performance of BRP and NCF are relatively poor. It verifies that considering user-item interactions independently is not effective for sequential recommendation. By considering the successive sequential dependencies, FPMC performs better than BPR and NCF.

For the neural models, we see that by adopting deep learning to capture sequential dependencies, these models obtain a better performance than conventional methods. Comparing with GRU4Rec, Caser further considers the union and skip behaviors of sequential patterns, and performs better than the former.

By utilizing an attention mechanism to highlight the most relevant items, we also find that the attention-based sequential models perform better than the none attention-based ones. This observation demonstrates the necessity of discerning the relevance between items for a better recommendation. An interesting observation is that HRL performs better than SASRec, it demonstrates dropping irrelevant items directly is a more effective strategy than assigning smaller weights on these items. Similarly, when removing irrelevant items according to a threshold, SASRec+ performs better than HRL and SASRec. For BERT4Rec, it employs deep bidirectional self-attention and beats all the other baselines on four datasets. Comparing with the attention-based sequential models and the other baselines, our proposed RAP achieves significantly better recommendation performance across the four datasets and two metrics. We believe that the sequence denoising devised in RAP is effective to enhance recommendation performance.

5.3 Analysis on RAP

In this section, we conduct experiments on RAP to gain a further insightful understanding. Specifically, we consider three variants as follows:

• RAP+p -c : we retain sequential patterns, and remove the contrastive learning strategy by setting r- to 0;
• RAP-p +c : we keep the contrastive learning strategy and remove sequential patterns from RAP;
• RAP-p -c : We remove both sequential patterns and the contrastive learning strategy from RAP;

Table 3 reports the performance comparison between RAP and its three sub-models on four datasets. We observe that RAP-p -c obtains the worst performance over all four datasets. It demonstrates making sequence denoising is not an easy task. There is no consistent dominant between RAP+p -c , and RAP-p +c , it indicates both fusing sequential patterns and using contrastive learning process complements each other for better preference learning. Obviously, RAP obtains the best performance against these variants on four datasets.

5.4 Qualitative Analysis

In our model, a major novelty is that we formalize the sequence denoising problem into a MDP. To obtain a better understanding why RAP performs better than other models, shown in Figure 2, we further construct a qualitative analysis with a case study on Beauty dataset. Specifically, we present a snapshot of the interaction sequence for a sampled user, which contains eight items. The first five items are relevant with face care (lipstick, moisturizer and sunscreen) and hair care (comb and shampoo) respectively. Given the first five items, we use RAP and BERT4Rec to recommend the next three items sequentially. For a convenient demonstration, we only retain the Top-1 recommendation results of two models.

As we can see, both RAP and BERT4Rec recommend the first item conditioner correctly, and they both assign higher weights on comb and shampoo, which are all relevant with hair care. However, BERT4Rec fails to make a correct recommendation on the next two items. The reason is that BERT4Rec is based on the MC assumption, hence the last visited item plays a key role. This makes BERT4Rec believe that the user still prefers items related with hair care, resulting in higher attention weights on these items (shampoo, conditioner and styler), leading to the wrong recommendation. On the contrary, RAP can well extract the truly sequential dependencies relevant with the target item, and recommends face mask and facial cleanser, which are relevant with face care. That is, RAP can make a more precise preference inference over the sequence dependencies from a more clean subsequence.

6 Conclusion

In this paper, we formulate a sequence denoising problem, and we propose a pattern-enhanced contrastive policy learning network for denoising and recommendation. RAP casts the sequential denoising problem as a form of MDP, and exploits sequential patterns and a contrastive learning process for preference learning. Extensive experiments show that RAP achieves state-of-the-art performance against a series of SOTA solutions.

Right now, we only utilize item interactions to perform denoising. This limited resources would hinder the effective learning of denoising and recommendation. In future, we will choose to extract sequential dependencies from the knowledge graph, which may bring further benefit towards the denoising process.

Acknowledgements

This research work was supported by fundamental Research for the National Natural Science Foundation of China(No.61802029,61872278). We would like to thank for the anonymous reviewers for their valuable comments.