Intent Preference Decoupling for User Representation on Online Recommender System

1 Introduction

One of the most challenging problems in online recommender systems is to predict users' current interests. Unlike search engines, where one could directly convey precise interests with queries like "black Nike sneaker" or "ionic hairdryer", the user's real-time interest in online recommender systems must be predicted from user history behaviors.

In general, there are two major types of factors affecting whether a user could be interested in an item: i) User intent factors imply users' current needs. Although recommender systems typically do not require the user to type the query words, most users visit a recommender with a certain degree of intent rather than completely wandering around. For example, when a user opens Netflix, (s)he may have already decided to watch a comedy or a documentary. For most users, they are only interested in a few categories during a certain visit. Such kinds of factors often change dynamically over time. Each visit user may come with a completely different intent. ii) User preference factors represent the utility of buying or clicking items displayed by the recommender system. The user's preference factor is not universal-each user has his(her) own preference on different items. For example, having the intent of buying a sneaker does not mean a user will click/buy every sneaker displayed by the recommender system; instead, the user still needs to consider its brand preference, shoe size, or price preference. Although user preferences could still be changing over time, those factors are generally stable and evolve in a gradual manner.

∗Corresponding Author

Most existing works on online recommender systems model users' intent and preferences as users' evolving "interests" without distinction, where typical online learning (or sequential learning) algorithms [Wang et al., 2018; Ying et al., 2018] are applied to capture changes in interests. Jointly learning intent and preference from users' historical behaviors in this way could be challenging, and the precision may suffer for two reasons. On one hand, typical online learning algorithms expect data distribution to be stable or change gradually [Hoi et al., 2018], while a user's intention could change dramatically from visit to visit. And thus, adapting the intent estimation from past intent could be inefficient in comparison to learning the intent factors separately for each new visit. On the other hand, user preference factors often require continuous learning over a long period. Dramatically changed user intent, however, may lead to catastrophic forgetting in learning user preference factors, and thus, preference factors could be much better learnt from a stable data distribution.

In this paper, we propose a learning framework to model user intent and preference explicitly. Different from previous online recommender systems which treat the learning of user interest as an online learning problem [Guo et al., 2019], our learning framework models the user interests in two spaces:

1. Intent Space. The learning process of intent embedding space is treated as a meta-learning problem instead of an online learning problem. Each session (which identifies a user visit) is viewed as an individual subtask. The metalearner is optimized to solve new learning tasks accurately

• using session samples. The intent embedding is learnt only from samples of the current session.
• Preference Space. The preference embedding space is learnt via online learning procedure based on samples with calibrated user intent. The variation of data distribution we aim to reduce is mainly due to the inaccurate estimation of user intent. Intuitively, if we know the user has intent on buying mobile-phone, an observation of unclicked Nike sneaker item could be better attributed to the intent of "not buying sneaker" but not the preference of "not like Nike"; in contrast, if the user has intent on buying sneaker, such an observation could be strongly attributed to the preference of "not like Nike" or some related price preference. If every data sample is accompanied with correct user intent, the variation of the learning process due to the sudden change of user intent is significantly reduced. Since user intent usually does not change during a session, we use the learnt user intent at the end of each session as the intent embedding for all samples in the session when learning the preference embedding.

As such, our proposed framework has two major advantages: 1) The user intent factors can be learnt much faster in a meta-learning fashion which is designed to learn from samples within the current session. 2) The user preference factors could benefit from the stable continuous learning process and be updated smoothly. Experimental results on two public datasets and a real-world recommender system proved its effectiveness. Except for the significant performance improvements over competitive online learning strategies, it provides a seamless way to combine with a variety of modern recommendation methods, including neural collaborative filtering [He et al., 2017] and sequential recommendation models [Hidasi et al., 2016; Zhou et al., 2018].

2 RelatedWork

User Modeling in Recommendation System. A series of works have discussed how to learn a better user representation to deliver personalized recommendation outcomes. Earlier user-based collaborative filtering represented each user by the row vector in a rating matrix [Herlocker et al., 1999] and recommended items according to the similar users. Matrix factorization [Paterek, 2007] firstly proposed to jointly map both user and item into a latent space of which each dimension reflects a certain user preference towards the item factor. Deep sequential models enrich the user representation by a series of model innovations like recurrent neural network [Hidasi et al., 2016], attention function [Zhou et al., 2018], and memory network [Chen et al., 2018]. Another line to model user historical behaviors is to study how to decouple the short-term and long-term preference [Ying et al., 2018]. However, most of them designed the delicate model structures under the offline settings. Our work followed this line but proposed an orthogonal view from the online learning optimization, which is more practical for a real-world recommender system.

Online Learning for Recommendation. A practical solution to solve the streaming feedback of large-scale recommender systems is to adopt online learning strategies [Jugovac et al., 2018], which attempt to capture the user's instant interest by updating the model with the new arrival data. Various online learning strategies have been explored for neighbor-based [Huang et al., 2015], graph-based [He et al., 2015], probabilistic [Chang et al., 2017], and matrix factorization method [He et al., 2016]. For example, [Chang et al., 2017] proposed a variational Bayesian approach to permit efficient instantaneous online inference. [He et al., 2016] adapted ALS optimization techniques for matrix factorization. As for deep learning-based methods, a natural choice to update parameters by gradient descent [Duchi et al., 2011]. However, all these proposed methods are tailored for the specific problem formulation like graph, matrix, or deep neural network. They more or less ignored the nature of intent factors and preference factors for user recommendation, just adapting the user intent instantly while encountering with the risk of forgetting the long-term preference since only the most recent data is used to update the model.

Meta Learning. Meta-learning has proposed a "learning to learn" framework, intending to learn an update rule or optimize for fast adaptation with a few training samples [Andrychowicz et al., 2016; Nichol et al., 2018]. While a few meta-learning works have discussed the online learning setting at meta-test time, nearly all previous algorithms assume that the task distribution is stationary during the metatrain stage. [Finn et al., 2019] introduced an online metasetting and extended the MAML algorithm [Finn et al., 2017] to deal with the changing task distributions.

When applying meta-learning for the recommendation, previous works have drawn the analogy that the estimation for different users' preferences could be taken as different subtasks and issued the cold-start problem for newly arrival users or items [Vartak et al., 2017; Lee et al., 2019]. However, they still considered it under the offline setting and thus ignored the streaming nature of a real-world online recommender system. [Du et al., 2019] considered an online metalearning based recommender system for estimating user preferences under different recommendation scenarios, while the parameter update overheads are heavy and hard to satisfy the requirement of online latency.

3 ProblemFormulation

Let Ut ⊆ U and It ⊆ I denote a set of users and a set of items at the time t, where |Ut| and |It| are the numbers of users and items, respectively. For each user u ∈ U, his/her temporally ordered sessions are denoted by Lu = {Su 1 , . . . , S u t , . . . , S u Tu } where Tu is the total number of sessions for user u and Su t = [sut,1 , sut,2 , . . . ], t ∈ [1, Tu] represents a list of successive interacted items at time stamp t. Under the meta learning setting, estimating user intent in each session browsing Su t is considered as a new task. Given users, their current browsing session Su t , and previous interacted sessions Lut-1 = {Su 1 , . . . , S u t-1 }, we aim at delivering the real-time recommendation results by the fast adaptive user intent learnt from Su t in a meta-learning manner and the user preference calibrated from Lut-1 . The notations used in this paper are summarized in Table 1.

4 Methodology

Since our proposed online learning strategy is orthogonal to the model structure of the recommender system, we first propose a unified framework for diverse recommendation models, with a particular focus on matrix factorization techniques and deep neural networks. The whole framework has three essential components: user modeling P(u), item representation Q(i), and user-item interaction strategy f. The prediction of the future ratings or behaviors is conducted as follows:

For matrix factorization, we project both users and items to a joint low-dimensional space, such that user-item interaction is modeled as inner product operation in that space. The formal definition can be specified as:

where pu and qi are latent factors for the given user u and item i, respectively. As shown in Figure 1.a, when adapting it into the neural network version, the interaction strategy f is alternated by multiple fully connected layers as follows:

where φ is the non-linearity activation function. Wl and bl are the corresponding parameters of the l-th layer.

Table 1: Notations.

Except for latent factors of users and items, sequential modeling is also involved in recently proposed recommendation methods [Hidasi et al., 2016; Zhou et al., 2018], which utilize the historical behaviors to enrich the representation of the user. The general formulation can be defined as:

where ⊕ denotes the concatenate operation, Lut-1 is the user historical behaviors until the time t and h(·) is the sequential modeling function, which could be average pooling [Covington et al., 2016], recurrent neural network [Hidasi et al., 2016], or attention function [Zhou et al., 2018].

4.1 Intent &Preference Decoupling

Under such a unified recommendation framework, a key component is the modeling of user representations P(u), which is directly related to learning the intent factors and preference factors for users. When resorting to the online learning framework, we propose to decouple the learning of user preference and instant intent from the user modeling P(u). Different from decoupling these two factors from behavior sequence [Ying et al., 2018], we separately model these two factors using the parameters pu of users. The reasons are as follows: 1) Under the online setting, we cannot maintain the lifelong behaviors of user histories due to the limitation of the storage overhead in the online system while the latent factor pu is updated all the time. The lifelong interaction trajectories are memorized in the representation of pu. 2) The parameter pu is personalized for different users. Compared with updating the model parameters during the online process, pu is expressive enough and much easier to be updated instantly.

As such, we maintain two sets of parameters (ps, pl) for users in parallel: ps is initialized by the meta-learner and responsible for adapting the instant intent of the current browsing, named as the intent embedding; pl is the preference embedding undertaking the role of learning a stable preference excluding the effects of changing user intent. After this decoupling, the model structure is shown in Figure 1.c and the prediction is computed by:

Algorithm 1 Follow the Intent and Preference Decoupling Require: Initial intent parameters θs 0 = {ps}, preference parameters θl0 = {pl, qi,W, b,W(h)}, and corresponding optimizers A = (As , Al ), objective function L Require: meta-learning rate η, meta-update number nmeta and frequency δ 1: randomly initialize w0 with the same dimensionality as θs 0 2: initialize the session buffer as empty, B ← [] 3: for t = 1, 2, . . . do 4: add B ← B + [Tt] // append task index into the buffer 5: given the task (session) data St at the time t 6: θs t ← INTENT-UPDATE(wt-1, θlt-1 , St, As , L) 7: θlt ← PREFERENCE-UPDATE(θs t , θlt-1 , St, Al , L) 8: if t mod δ = 0 then 9: wt = META-UPDATE(wt-1, B, t, η, nmeta) 10: B ← [] 11: else 12: wt ← wt-1 13: return parameters θl ,wt

Wh]. The set of trainable parameters is θ = {W, b, ps, pl, qi, W(h)} where W(h) is the weight collection of h.

Until now, we could describe our Follow the Intent and Preference Decoupling (FLIP) strategy with this decoupled model design and discuss its properties. FLIP mainly consists of two essential components: 1) session intent learning from the meta-learnt initialization value; 2) preference learning after the calibration of the user intent. In the following subsections, we will introduce the details of these two components.

Learning of User Intent

As mentioned in section 1, capturing the user browsing intent is an extremely challenging task since the user intent may change from visit to visit. Even in the same browsing session, the limited interaction behaviors pose great challenges on intent modeling. Meta-learning, also called learning to learn, aims to train a model that can rapidly adapt to a new task that does not appear during the training with a few examples. From this inspiration, we propose a meta-based recommender system that can rapidly estimate a new session browsing intent based on only a few interactions. By adopting the Online MAML algorithm [Finn et al., 2019], we can reformulate the problem as: 1) estimating user intent in each session browsing Su k is regarded as a new task drawn from the current task distribution Pt(T ); 2) the first m behaviors in Su k are taken as the support set for intent learning and denoted by Dtr k while the rest is the query set for intent testing and denoted by Dval k .

Instead of learning the whole parameters θ of the model in a meta manner, we consider the interaction strategy f is stable among different browsing sessions. Thus the meta-learnt parameter is focused on the intent embedding ps of the user. We attempt to find a good initialization value w of user intent parameter ps such that after a few update steps on ps from w by support set Dtr , the model can quickly learn an intent rep-

Algorithm 2 FLIP Subroutines

1: function INTENT-UPDATE(w, θl , St,A,L) 2: Synchronize parameter θs 0 ← w, n = |St| 3: for i = 1, . . . , n steps do 4: Record the performance of st,i by parameters θs i-1 , θl 5: θs i = θs i-1-A(L, θs i-1 , θl , st,i) // meta test, intent embedding update 6: return θsn 7: function PREFERENCE-UPDATE(θs , θl , St,A,L) 8: Perform update θl = θl -A(L, θs , θl , St) by Eq. (7). 9: return θl 10: function META-UPDATE(w, B, t, η, nmeta) 11: for i = 1, . . . , nmeta steps do 12: Sample task Tk from B 13: Dtr k ← the first m interactions of Sk, Dval k ← Sk - Dtr k 14: Compute gradient gt(w) using Dtr k ,Dval k , and Eq. (6) 15: Update parameters w ← w - ηgt(w) 16: return w

resentation and generalize well to the rest behaviors in Dval .

Specifically, at each meta-train stage, we collect a set of session behaviors {S1, . . . , SK} from different users drawn from Pt(T ). For simplicity, we omit the superscript of Su k since we do not distinguish the sessions from different users and just take each session as a subtask. The update of the initialization value w to adapt for the new task distribution is computed as follows:

where L is the loss function, α is the hyper-parameter to control the learning step size, and Uk(w) is the updated parameter of w for the task k using the support set Dtr k from the corresponding session behaviors. gt(w) is the aggregated gradient direction with respect to the generalization loss among different sessions from the task distribution Pt(T ). The learning procedure is abstracted as function META-UPDATE in Algorithm 2.

At meta-test time, the user intent parameter ps is initialized by the meta-learnt parameter wt-1 at time t - 1. As illustrated in function INTENT-UPDATE, we perform parameter update on ps by predefined loss function L and optimizer As , which could be stochastic gradient descent [Zinkevich, 2003] or other advanced online learning optimizers like Adagrad [Duchi et al., 2011]. The rest parameters θ/ps is fixed during this intent learning process and we record the testing performance for each interaction.

Learning of User Preference

For recommendation, the prediction errors are generally from two aspects: 1) the inaccurate estimation of the user intent which caused a significant deviation of the recommended items; 2) the forgetting of user preference which caused a mismatch between the recommended item features and user profile. To separate these two kinds of errors, we proposed to iterate incoming session behaviors twice for the intent learning and preference learning, respectively. The first pass is to calibrate instant intent as illustrated in function INTENTUPDATE. The second pass is to learn a relatively stable preference representation based on the calibrated user intent. Formally, the learning of the user preference is optimized as follows:

where the rest parameters {pl, qi, W, b, W(h)} are all updated in this pass and β is the hyper-parameter to control update step size. It performs like replaying the session game and fixing the rest errors of the last time. ps is fixed during this preference learning process as the intent representation.

To sum it up, our proposed Follow the Intent and Preference Decoupling (FLIP) strategy is presented in Algorithm 1. We first initialize a task buffer B = [], which is designed to maintain the most recent δ incoming sessions. When presented with a new session task at time t, we add task Tt to B and retrieve the corresponding session data St. In the subroutine, we first adapt the user intent embedding ps from wt-1 using the session data St by INTENT-UPDATE. After that, we reiterate the data and update the preference parameters in PREFERENCE-UPDATE. For every δ iterations, we update the initialization value w by META-UPDATE to adapt for the most recent task distribution.

5 Experiments

In this section, we conduct experiments to answer the following questions:

RQ1: does our proposed FLIP learning strategy provide a general solution for current mainstream recommendation models and further improve recommendation performance?

RQ2: what's the influence of instant intent and long-term preference in our learning strategy?

RQ3: which parameters are of vital importance for the proposed learning strategy and how do they affect the online learning performance?

5.1 Experimental Settings

Datasets. 1) MovieLens 1 is a widely used dataset to evaluate collaborative filtering algorithms. We adopted the version containing one million ratings from 6040 users and transformed it into a binary class dataset where the ratings above 3.5 are labeled as the positive samples while the rest are the negative ones. For each user, we sorted the interactions in chronological order and treated the behaviors during one day as a session. 2) Advertising 2 is a public dataset released from Tianchi Competition 3 by Alimama, ranging from 2017-05-06 to 2017-05-12. We filter out the items or users of which the interaction number is less than 5 and obtain 23.7 million ad display/click records from 0.67 million users and 0.38 million ads. The clicked and unclicked ones constitute the positive and negative samples, respectively. Following [Grbovic and Cheng, 2018], we treated the behaviors during 30 minutes as a session and obtained a total of 4,914,360 sessions. On average, each user had 7.29 session records. 3) Recommender is a sampled real-world recommendation dataset from a well-known e-commerce platform. The dataset is collected from 2019-12-01 to 2019-12-15 where each day has 10 billion display/click logs from 30 million users and 60 million items. We randomly sampled 1/100 users for experiments and segmented the behaviors into sessions for each user in the same way as Advertising dataset. The clicked and unclicked items are the corresponding positive and negative feedback for model training. On average, each user had 9.66 session interactions. The details of dataset statistics are listed in Table 3.

1 https://grouplens.org/datasets/movielens/1m/

2 https://tianchi.aliyun.com/dataset/dataDetail?dataId=56

3 https://tianchi.aliyun.com/home/?lang=en-us

Competitors. We compared our proposed FLIP learning strategy with the following representative methods.

• OGD & OGD' [Zinkevich, 2003]. OGD is a standard online learning strategy by gradient descent that updates model parameters with the most recent data samples.
• RMFX [Diaz-Aviles et al., 2012] is a modified matrix factorization method designed for streaming social recommendation. We adopt the sampling strategy of the paper and allow the model to update the parameters according to the most recent session behaviors and randomly sampled historical sessions from users simultaneously.
• SSRM [Guo et al., 2019] is a proposed streaming session recommendation model that tackled the problem of uncertainty of user behaviors in the session and highvelocity nature of streaming data. The learning strategy is also based on OGD, while an active sampling strategy is proposed to retrieve the historical sessions for model updates. Specifically, we compared our method with the sampling strategy of SSRM.

Except for the compared learning strategies, the experimented recommendation methods are listed as follows:

• NCF [He et al., 2017] is a matrix factorization method that alternates the interaction strategy between user and item by fully connected layers for high-order correlations.
• DNN [Covington et al., 2016] is an extension for NCF that involves the user's historical behaviors by average pooling operation.
• GRU4Rec [Hidasi et al., 2016] is a session-based recommendation method using gated recurrent neural network to compress intra-session interactions. We extended it to model multi-session behaviors and explicitly added the input of latent factors for users.
• uDin [Zhou et al., 2018] is a modified user-based attention model where the query of the attention function for historical behaviors is the latent factor of the user rather than the target item.

Metrics. AUC (Area Under ROC Curve) is a widely used metric to measure the goodness of the order by ranking all the items according to the predicted probabilities. Despite its popularity and effectiveness, the detail ranking performance for a specific user or session browsing is hard to reveal. [He and McAuley, 2016; Zhou et al., 2018] proposed a variation of user weighted AUC which measures the goodness of intrauser order by averaging AUC over users, and we further refine it for the session browsing. It is calculated by:

(a) MovieLens.

where n is the number of sessions, #displaysi is the number of displayed items, and AUCi is the corresponding AUC metric of the i-th session browsing. Normalized Discounted Cumulative Gain (NDCG@k) is also adopted here to measure the ranking performance of the clicked item at the top k positions. Without a specific statement, the k is set to 5.

5.2 Overall Performance (RQ1)

Table 2 summarized the experimental results and we have the following key observations: 1) Compared with existing online learning methods, FLIP learning strategy achieves the best performance in all three datasets. From the results between OGD and OGD', it verifies the indispensable importance of modeling user latent factors explicitly and proves the rationality of designed decoupling strategy from the latent factors. Beyond that, as in RMFX and SSRM, designing sampling strategies is also a feasible solution to retrace the user preference in previous interacted sessions. However, the learning of the intent and preference is still mixed up and the improvements in session-based metrics are marginal. 2) The benefits brought by decoupling the learning of user intent and preference are versatile among our experimented recommendation methods. The improvements are significant for the user-based deep attention network (uDin) since we could employ the intent embedding and the preference embedding as the query to attend different information from historical behaviors. 3) Compared with the improvements on AUC metric that measures the overall ranking performance without the distinction of users or sessions, we focus more on session-based metrics pAUC and NDCG@5. As a whole, the improvements on pAUC range from 0.43% to 1.49% where the 0.3% improvements are significant due to the large scale of the testing dataset where we could consider the whole training set is also the testing set during the online update; meanwhile, the NDCG@5 is improved by a large margin, ranging from 1.02%-20.3%. 4) The extent of the improvements changes among different datasets. Compared with MovieLens dataset, Advertising and Recommender datasets are much harder since the user browsing intent changes more frequently on the e-commerce platform. Most of the time, the browsing behavior in the e-commerce platform is driven by the specific purchasing demand, which is hard to predict. Under such circumstance, our proposed method tailored for user intent and preference decoupling achieve a large improvement for these two datasets on session-based metrics, i.e. NDCG@5.

5.3 Influence of Components (RQ2)

To help understand the functionality of each component of our proposed learning strategy, we experiment on two variants of FLIP. 1) FLIP-Z is a simplified version that during each session, the intent embedding is initialized by zero vector and updated in the session browsing. The preference parameters are updated after the calibration of the intent. 2) FLIP-S is the variant that updates the intent and preference parameters simultaneously where the intent embedding is initialized in a meta manner. Due to the limitation of space, we only show the results of NCF model in Figure 2 and the results of the other three methods admit the same trend thus they are omitted. We observed that the performance variation is different among datasets. For MovieLens dataset, FLIP-S performs slightly worse than FLIP-Z variant. Generally, user interests in movies are evolving gradually. In FLIP-S, the intent embedding and preference embedding are all updated instantly with new arrival data, which caused that the learning of a relatively stable preference representation is hindered by such instant updates. In contrast, the learning of the intent is more important for Advertising and Recommendation datasets and we observe the better performance on FLIP-S over FLIP-Z and OGD, which further verifies the necessity of meta-learning mechanism on session intent learning.

5.4 Influence of Hyper-parameters (RQ3)

We aim at pointing out which hyper-parameters affect our proposed FLIP learning strategy in this section. The analyzed hyper-parameters including the dimensionality of the intent embedding and the update frequency δ of META-UPDATE procedure. Specifically, we set the dimensionality of intent embedding as 1/8, 1/4, 1/2, and 1 proportional to that of preference embedding which we set to 64 in our experiments. As shown in Table 4, FLIP learning strategy is effective under different dimensionality settings. For MovieLens and Advertising datasets, we observed a 1/4 dimensionality is sufficient for learning user instant intent since MovieLens is a relatively small dataset and the number of intra-session behaviors is relatively sparse for Advertising dataset. As for Recommender datasets, a larger-sized dimensionality could further achieve better performance as more and more session browsing behaviors are introduced for model training.

Considering the characteristics of three datasets, we designed different hyper-parameter settings on update frequency δ of META-UPDATE procedure. As shown in Table 5, shorter update interval can timely capture the drift of user interests and lead to better intra-session performance. We observed a one-month time interval is appropriate for MovieLens dataset since user interests do not change too much during this time. User behaviors in advertising and recommender are always high velocity and huge volume. We find a 30-minutes or 1-hour setting is more suitable.

6 Conclusion

In this paper, we designed an online learning strategy to decouple the learning of the user intent and preference for online recommender systems. By abstracting each session browsing as a meta-learning task, we realized the fast adaptation when learning user instant intention. The preference learning is then conducted, accompanied by the calibrated session behaviors. With this two-phase learning strategy, the intent and preference factors of user representation are effectively modeled and decoupled during the online learning process. We validated the effectiveness of our proposed learning strategy on two public datasets and a real-world online recommender system. The experimental results revealed the advantages of our proposed learning strategy over competitive baselines.

Acknowledgments

We thank anonymous reviewers for their insightful and helpful comments, which improve the paper. Thanks for the discussions with the colleagues, Chen Xu, Junfeng Ge and Yaliang Li who provide a lot of insights for the paper work. Thanks for the hard work of the co-authors who are devoted in this work.