Improving Sequential Recommendation Consistency with Self-Supervised Imitation

1 Introduction

With the development of mobile devices and Internet services, the past few years have seen the prosperity of a broad spectrum of recommendation systems (RS). It facilitates individuals making decisions over innumerable choices on the web. RS attracts a growing number of online retailers and e-commerce platforms to meet diverse user demands, enrich and promote their online shopping experiences.

� Work done at JD.com.

In real-world applications, users' current interests are affected by their historical behaviors. When a user orders a smartphone, he/she will subsequently choose and purchase accessories like chargers, mobile phone covers, etc. Such sequential user-item dependencies prevail and motivate the rising of sequential recommendation, where the user history interaction sequence is treated as a dynamic sequence. And the sequential dependencies are taken into account to characterize the current user preference to make more accurate recommendations[Chen et al., 2018].

As for the sequential recommendation, a surge of methods have been proposed to capture the sequential dynamics within user historical behaviors and predict the next use-interested item(s): Markov Chains[Zimdars et al., 2001]recurrent neural networks (RNNs)[Hidasi et al., 2016b], convolutional neural networks (CNNs)[Tang and Wang, 2018; Yuan et al., 2019], Graph Neural Network(GNN)[Wu et al., 2019], self-attention mechanisms[Ying et al., 2018]. Markov chain-based models adopt K-order user-item interaction sequential transitions. CNN handles the transition within a sliding window, whereas RNN-based methods apply GRU or LSTM to compress dynamic user interests with hidden states. GNN-based methods take advantage of directed graphs to model complex useritem transitions in structured relation datasets. And selfattention mechanisms emphasize relevant and essential interactions with di�erent historical user-item interaction weights.

Though previous methods show promising results, current sequential recommendation system learning relies heavily on the observed user-item action prediction. As a result, their representations and expressiveness are still limited. Such representation learning signal is too sparse to train expressive-enough item representations [Rendle et al., 2010; Song et al., 2019]. Not to mention that those subtle but diverse persona di�erences among users are also neglected. In this paper, we attempt to abstract the consistency knowledge with self-supervised pre-training tasks by exploring three aspects of consistency. To handle user-interaction dynamics, we propose to abstract consistency information with temporal and persona consistency, where temporal consistency reflects that the recommender is expected to organize and display the items in a proper order to satisfy users' interests. Persona consistency stands on the fact that the recommender should be capable of perceiving those diverse persona distinctions from the user-item interaction sequence.

Temporal and persona consistency knowledge is quite a straightforward approach to improve sequential recommendation systems by modeling instant interaction dynamics. However, long-term global session consistency is overlooked. Without a global perspective, the model still su�ers from noisy actions and makes inconsistent predictions. A typical case is that when a user unconsciously clicks wrong items, the system will be easily a�ected by short-term click and instantly make unrelated predictions. To mitigate such defect, we further bring global session consistency into the sequential recommendation, which enhances the item representations by maximizing the mutual information between global and local parts of the interaction sequence.

Though self-supervised consistency knowledge can be quite beneficial, directly optimizing the consistency is intractable. It's because training is performed on the whole session, which is not attainable in inference. Furthermore, comprehensively integrating all three aspects of consistency pre-training knowledge plays a vital role in estimating user's preferences more accurately and consistently. Therefore, we propose an integrated imitation learning framework. Consistency-enhanced pre-training models serve as teachers, and a sequential recommendation model is treated as a student. The consistency knowledge is e�ectively internalized and transferred to the student model by imitating the conventional prediction logit as well as the consistencyenhanced item representations. Additionally, with the flexibility of Self-Supervised Imitation (SSI) framework, the selfsupervised learning can be easily transferred to other student recommenders as demand.

Experiments on four real-world datasets show that our selfsupervised imitation framework e�ectively outperforms several state-of-the-art baselines.

2 Methodology

2.1 Problem Statement

Suppose I is the set of items and U is the set of users. For a specific user u 2 U, list Iu = [q (u) 1 ; q (u) 2 :::; q (u) k ] denote the corresponding interaction sequence in chronological order where qk 2 I and k is the time step. Given the interaction history Iu, sequential recommendation aims to predict the item that user u will interact with at time step k + 1.

2.2 Teacher Base Model

Our teacher base model is a BERT-based structure[Sun et al., 2019], built upon the popular self-attention layer. The model randomly masks items in the user history interaction sequence in the training phase and replaces them with a unique token [MASK]. Then predicts the original ids of the masked items based on the context. In the test phase, the model appends the special token [MASK] at the end of the input sequence and then predicts the next item based on the final hidden representation of this token.

2.3 Self-supervised Consistency Pre-training

Existing studies[Hidasi et al., 2016b; Hidasi et al., 2016a; Kang and McAuley, 2018; Logeswaran and Lee, 2018]

(c) Global Session Consistency Task

Figure 1: Self-supervised consistency pre-training tasks. (a) The order of some n-grams is shu�ed in the sequence for the chronological prediction task. (b) Some n-grams in the sequence are replaced with other users' interaction n-grams. (c) The mutual information is maximized between some n-grams and the rest of the sequence.

mainly emphasize the e�ect of sequential characteristics using user-item action prediction objective alone. Solely relying on such sparse learning signals may result in underlearned item representations. We enrich learning representations' expressiveness by extracting consistency knowledge with self-supervised learning objectives to mitigate such defects. As exemplified in Figure 1, we introduce three aspects of recommendation consistency: temporal, persona and global session consistency, respectively, where temporal consistency and persona consistency capture user-interaction dynamics and global session consistency enhances the model with the global perspective.

Temporal Consistency

Temporal consistency captures users' interaction sequence so that the recommendation system can display the items in a proper order that will satisfy users' interests better. Therefore, we design a user-item interaction sequence chronological order recognition task to enhance the model temporal sensitivity. First, we draw positive and negative samples by randomly exchanging the order of some n-gram sub-sequences in the user's interaction history with Bernoulli distribution. Then we add a unique token named [INT] at the end of the sequence and let the corresponding hidden representation go through a classifier to predict whether the interaction sequence is in the original order or not. The loss of temporal consistency is formulated as:

where D is the set of input, x t i is the output embedding, y t i is label, Ltemp is calculated by cross-entropy, and h t (�) is a MLP.

Persona Consistency

Person consistency models the diverse persona distinctions among di�erent users. Positive samples are the interactions from one user. In contrast, for negative samples, we replace some user-item interaction n-grams with other items. Similarly, we add a unique token [INT] at the end of the sequence, and a classifier predicts whether the user will interact with the current sequence. Persona consistency knowledge is perceived by di�erentiating whether the given user-item sequence is from one user. The loss of persona consistency is formulated as:

where D is the set of input, x p i is the output embedding, y p i is label, Lpers is calculated by cross-entropy, and h p (�) is a MLP.

Global Session Consistency

Researches on image representation learning show that mutual information maximization between an image representation and local regions improves the quality of the representation[Hjelm et al., 2019]. Similar observations are also found in sentence representation learning [Kong et al., 2019]. Inspired by previous studies, to avoid noise and provide the model with a global perspective, we further introduce the global session consistency as a self-supervision task for the sequential recommendation. Given a user interaction sequence I = [q1; q2:::; qk], we consider the local representation to be the encoded representations of n-grams in the sequence. The global representation is the rest of the masked sequence representation, which corresponds to the last token hidden state. We maximize the mutual information between the global representation and local representation. Denote an n-gram by qn:m and a masked sequence at position n to m by ˆqn:m. We define Lglobal as:

where ˜qn:m is an n-gram from a set ˜ S that consists of the positive sample ˆqn:m and negative n-grams from other sequence in the same batch, d(�) is the base model.

Consistency Pre-Training

To ensure that each aspect of consistency is well-learned, we establish three independent models upon BERT4Rec[Sun et al., 2019] to induce temporal consistency, persona consistency, and global consistency information. The training objectives are defined as:

where LMLM is the loss of the base model, L i I is the loss of each model. Ltemp, Lpers and Lglobal are augmented as a kind of regularization to abstract the consistency knowledge from the sequence. �i is a hyper-parameter balancing the importance of the two losses.

2.4 Self-supervised Imitation Learning

So far, three aspects of consistency knowledge acquired from the proposed self-supervised learning tasks. However, sequential recommendation system still faces two major roadblocks in augmenting them: 1) in our approach, consistency learning is conducted on the entire user-item interaction session, whereas only the historical user behaviors are available during inference and future interactions are unattainable; 2) given three aspects of consistency-enhanced representations, it is paramount to explore a comprehensive integration of them to estimate a user's preference with better accuracy and consistency.

We propose an imitation learning framework to overcome the obstacles, where we treat the consistency-enhanced models as teachers and another sequential recommendation model as a student. The student is designated to distill consistency knowledge from three self-supervision-enhanced teachers by performing both the prediction distribution imitation and item representation imitation. Figure 2 illustrates the general selfsupervised imitation learning framework.

Prediction Distribution Imitation

In prediction distribution imitation, the student model distills the merits from the teacher model with prediction behaviors. In particular, we minimize the KL divergence of the predicted probability distribution between the teacher and the student:

where Q is the set of candidate items, �1 and �2 are the parameters of the teacher and student model, respectively.

Item Representation Imitation

Item representation imitation learns from the teacher by imitating the consistency-enhanced representations. In detail, we restrict the item representations discrepancy between the student and the teacher:

where g t and g s are the outputs of item representations in teacher and student respectively, f (�) is the mean-squarederror (MSE) loss.

Integrated Imitation Training

To e�ectively integrate three independent consistency teachers with the student, given the loss L 1 IL in Equation 5 and the loss L 2 IL in Equation 6, the final objective function of the student is formulated as:

where Ls is the loss of student model, �i is the teacher importance weight, n is the number of teachers. To further enable the model to control the e�ects of the teacher adaptively, we set �i to be learnable parameters that are optimized during training regularization:

where the initial value of �i is set to 1. The 1=�i and 1=� 2 i regularizations induce the model to learn to choose more effective teachers.

Note that, with the flexibility of the self-supervised imitation framework, consistency knowledge can be easily transferred to many student recommenders.

3 Experiments

3.1 Experimental Settings

Data Sets

We conduct our experiments on real-world datasets from Amazon review datasets.In this work, we select four subcategories: "Beauty", "Toys and Games", "Sports and Outdoors" and "Musical Instruments". We select users with at least five interaction records for experiments. The data statistics after preprocessing are listed in Table 1.

To evaluate the recommendation's performance, we split each dataset into training/validation/testing sets. We hold out the last two interactions as validation and test sets for each user, while the other interactions are used for training.

Evaluation Metrics

Usually, the recommendation system suggests a few items at once. The desired item should be amongst the first few listed items. Therefore, we employ Recall@N and NDCG@N [Li et al., 2013] to evaluate the recommendation performance. In general, higher metric values indicate better ranking accuracy. Recall@k indicates the proportion of cases when the rated item is amongst the top-k items. NDCG@k is the normalized discounted cumulative gain at k, which takes the rank of recommended items into account and assigns larger weights on higher positions. To avoid high computation cost on all user-items in evaluation, following the strategy in [Guo et al., 2019], we randomly draw 99 negative items that have not been engaged with the user and rank the ground-truth item among the 100 items.

Baselines

We compare with the following sequential recommendation models to justify the e�ectiveness of our approaches.

• � MostPop recommends the item according to the popularity measured by the number of user-interactions, which provides a simple non-personalized baseline.
• � GRU4Rec[Hidasi et al., 2016b] adopts GRU to capture sequential dependencies and makes predictions for session-based recommendation.
• � BERT4Rec[Sun et al., 2019] learns a bidirectional representation model to make recommendations by learning a cloze objective with reference to BERT.
• � HGN[Ma et al., 2019] captures both the long-term and short-term user interests by hierarchical gating network.
• � S 3 -Rec[Zhou et al., 2020] is a recently proposed selfsupervised learning framework under mutual information maximization principle for the sequential recommendation, which is based on self-attentive neural architecture.
• � GREC[Yuan et al., 2020] is an encoder-decoder framework that trains the encoder and decoder by a gap-filling mechanism.

Parameter Settings

For MostPop and GRU4Rec, we implement them with PyTorch. For S 3 -Rec, we use two auxiliary self-supervised objectives without extra attribute information to ensure fairness. For other methods, we use the source code provided by their authors. All hyper-parameters are set following the suggestions from the original papers. For SSI, our teacher model bases on BERT4Rec[Sun et al., 2019] and our student model bases on HGN[Ma et al., 2019]. We set the number of the self-attention blocks and the attention heads as 8 and 4. The dimension of the embedding is 256. The hyper-parameters are set as �1 = �2 = �3 = 1. We use the Adam optimizer[Kingma and Ba, 2015] with a learning rate of 0.001, where the batch size is set as 256 in the teacher and student model, respectively.

3.2 Results and Analysis

Overall Results

Table 2 presents the performance comparisons between several baselines and our model (SSI). SSI consistently achieves the best performance on four datasets with all evaluation metrics, verifying our model's superiority. Compared with BERT4Rec, our model shows much better performance. In our model, bidirectional item representations are further improved with three aspects of consistency. The integrated imitation network e�ectively distills consistency-enhanced pretraining knowledge into the student model, which combines the merits of the pre-training teachers and student network. SSI outperforms HGN. The reason is that HGN undertakes the student network in our model. The performance of HGN is e�ectively improved by imitating self-supervised consistency knowledge. The performance of S 3 -Rec is inferior to SSI. Although the absence of extra attribute information is a potential influencing factor, another primary reason is that S 3 - Rec ignores the consistency of sequential recommendation.

Flexibility of Self-supervised Imitation

In our self-supervised imitation framework, the student recommender can be any sequential recommendation system. To verify its flexibility, we also list the performance of GRU4Rec as the student model. In Table 2, we notice that GRU4Rec and HGN are both improved by self-supervised imitation (SSIGRU4Rec, SSI), which demonstrates its applicability. More encouragingly, concerning the performance of the base model, GRU4Rec is much inferior to HGN. However, after being augmented with self-supervised imitation, the performance of SSIGRU4Rec is on par with SSI. In conclusion, the self-supervised consistency knowledge can be easily transferred to other student recommenders with the flexibility of the self-supervised imitation framework.

Impact of Di�erent Aspects of Consistency

In this paper, we propose three aspects of self-supervised pretraining tasks to abstract consistency knowledge. Figure 3 shows the comparison of e�ects between the proposed three aspects of consistency, temporal, persona, and global session consistency. Compared with the base model, we observe that independently utilizing temporal, personal, or global session consistency e�ectively boots the model performance. The gains from di�erent aspects of consistency vary on di�erent datasets. However, jointly imitating all three aspects of consistency-enhanced knowledge achieves the best performance.

Impact of Pre-training Model Size

In our model, consistency pre-training is performed upon BERT4Rec. As self-attention based pre-training is computation-expensive, we would like to verify the model performance on di�erent pre-training model layers. Figure 4 illustrates the model performance comparison between our model and BERT4Rec along with di�erent self-attention layers. We observe the performance degradation of both models along with the decreased number of self-attention layers. What's more, the performance gap increases when comparing a relative deep model with eight self-attention layers and a shallow one with only one self-attention layer. Obviously, our model is more robust to computational cost than BERT4Rec. We conjecture that this is because our model effectively enriches the item representation expressiveness with three consistency-distillation objectives.

Impact of Integrated Imitation Learning

Integrated imitation learning transfers consistency-enhanced knowledge from three aspects of teachers to the student. In our framework, we employ the prediction logit imitation as well as the item representation imitation. Table 3 shows the performance di�erences. Both kinds of imitations contribute to the final performance. Though prediction distribution regularization is quite useful for consistency knowledge transferring, item representation imitation further improves the knowledge distillation e�ciency. We speculate that item representation provides a short-cut for consistency knowledge distillation.

4 RelatedWork

Nowadays, many approaches have been proposed to model the user's historical interaction sequence. The methods based on the Markov chain predict the subsequent user interaction by estimating the probability of transfer matrix between items[Zimdars et al., 2001]. RNN-based methods model the sequential dependencies over the given interactions from left to right and make recommendations based on this hidden representation. Except for the basic RNN, long short-termmemory (LSTM)[Wu et al., 2017], gated recurrent unit (GRU)[Hidasi et al., 2016b], hierarchical RNN[Quadrana et al., 2017] have also been developed to capture the long-term or more complex dependencies in a sequence. CNN-based methods first embed this historical interactive information into a matrix and then use CNN to treat the matrix as an image to learn its local features for subsequent recommendation[Tang and Wang, 2018; Yuan et al., 2019]. GNN-based methods first build a directed graph on the interaction sequence, then learn the embeddings of users or items on the graph to get more complex relations over the whole graph [Wu et al., 2019]. Attention models emphasize those important interactions in a sequence while downplaying those that have nothing to do with user interest[Ying et al., 2018].

Self-supervised learning[Lan et al., 2020; Devlin et al., 2018] prevails in language modelling. It allows us to mine knowledge from unlabeled data in a supervised manner. S 3 - Rec[Zhou et al., 2020] enhances data representations and learns the correlations with mutual information maximization for the sequential recommendation, whereas we enrich the item representation expressiveness with the temporal, person and global session consistency, and distills the consistency enhanced knowledge to the student by imitation learning.

Knowledge distillation[Hinton et al., 2015] introduces soft-target related to the teacher network as a part of the loss to induce student network training and realize knowledge transfer. Model compression[Buciluundefined et al., 2006] is a great application of knowledge distillation, which uses a light model to learn the knowledge of the heavy model to improve the e�ciency of inference. In our work, we combine the benefits of self-supervised learning and imitation learning for the sequential recommendation, where three elaborately designed self-supervised consistency learning tasks transfer knowledge through integrated imitation learning.

5 Conclusion

In this work, we improve sequential recommendation consistency with self-supervised imitation. First, three aspects of consistency knowledge are extracted with self-supervision tasks, where temporal and persona consistencies capture useritem dynamics, and the global session consistency provides a global perspective with interaction mutual information. Then an imitation framework integrates the consistency knowledge and transfers it to the student. Due to its special merits of flexibility, consistency knowledge can easily benefit other student recommenders as demand. Experimental results and analysis demonstrate the superiority of the proposed model.

Acknowledgments

This work was supported by National key R&D Program of China (Grant No. 2018YFB0904503). Hongshen Chen and Yonghao Song are the corresponding authors.