A Review-Driven Neural Model for Sequential Recommendation

1 Introduction

Recommender systems are now an indispensable asset in many E-Commerce platforms to enhance their productivity. It is well recognized that these systems largely drive sales and improve user satisfaction by automatically pushing what a user is looking for [Keiningham et al., 2007]. The next purchase decision made by a user is often influenced by her recent behaviors. For example, after buying a SLR camera, the user would be highly interested in camera lenses. Recently, sequential recommendation has drawn increasing attention from both academic and industrial circles. The task is to identify the next item that a user will purchase by considering her temporal preference as a sequence of purchased items.

The key challenge of sequential recommendation is to dynamically approximate the current preference of a user by considering both the general preferences and sequential patterns between items. As being the seminal work, FPMC extends the latent factor learning model by encoding the sequential transition patterns between items [Rendle et al., 2010]. The prediction made by FPMC is a linear combination of the user general preference and item sequential association which both are calculated by conducting inner-product between the corresponding latent vectors. Following this line, many variants are proposed to include more complex patterns [Wang et al., 2015; He and McAuley, 2016]. Recent advances in deep learning techniques have bred many neural solutions for sequential recommendation. Both convolutional neural networks (CNN) and recurrent neural networks (RNN) based models are proposed to capture a user's dynamic preferences [Hidasi et al., 2015; Yu et al., 2016; Chen et al., 2018; Tang and Wang, 2018], leading to state-ofthe-art recommendation performance. However, these existing methods do not consider user reviews to enhance sequential recommendation. The semantic information contained in user reviews can reveal different features of items and also the preferences of users. It is widely recognized that exploiting review information would largely improve the rating prediction accuracy [Wang and Blei, 2011; Bao et al., 2014; Zheng et al., 2017; Wu et al., 2018; Wu et al., 2019; Li et al., 2019].

∗Xiangyang Luo is the corresponding author.

To this end, we propose a review-driven neural model for sequential recommendation, named RNS. The incorporation of review text brings more rich semantic information and strengthens model's expressive ability. Specifically, we first embed users and items into low-dimensional dense spaces through aspect-aware convolutional processing of the review documents, at the same time, user's general preference is extracted. Then a hierarchical attention over attention mechanism is employed to capture user sequential pattern at both union-level and individual-level. After that, we combine user's intrinsic (long-term) preference and temporal (shortterm) preference to get a hybrid representation and make more accurate next items recommendation. On three realworld datasets from different domains, RNS outperforms the existing state-of-the-art methods remarkably. Overall, the key contributions of this work are summarized as below:

• We propose a novel review-driven neural model that exploits reviews for sequential recommendation. To the best of our knowledge, this is the first attempt to harness

the rich semantics from reviews for this task.

• We introduce an aspect-based convolutional network to identify user general preference from review document and utilize a hierarchical attention over attention mechanism to model user sequential preference at different granularities.
• On three real-world datasets with diverse characteristics, our results demonstrate the superiority of the proposed RNS in terms of four metrics.

2 RelatedWork

In the following, we review the existing literatures that are highly relevant to our work: namely review-based recommendation and sequential recommendation.

2.1 Review-based Recommendation

Conventionally, many recommender systems are mainly developed in the paradigm of collaborative filtering (CF). Within this line of literatures, great effort is made under the framework of matrix factorization or the probabilistic counterpart [Salakhutdinov and Mnih, 2007; Koren et al., 2009]. The idea is to represent each user or item as a low dimension latent vector. The binary interaction for a user-item pair is then estimated by performing inner-product of the corresponding latent vectors. However, the user-item interaction data is very sparse, which is an inherent obstacle for latent factor learning based solutions. Currently, to tackle the data sparsity problem, many techniques have been developed by utilizing the semantic signals provided by reviews. These advances have delivered state-of-the-art recommendation performance [Wang and Blei, 2011; Bao et al., 2014; Zheng et al., 2017; Catherine and Cohen, 2017; Wu et al., 2018; Wu et al., 2019; Li et al., 2019].

Many works propose to distill semantic latent factors from reviews with topic modeling techniques [Blei et al., 2003]. These solutions incorporate the topical factors learnt from reviews into latent factor learning framework [Wang and Blei, 2011; Bao et al., 2014]. However, given that the topic modeling represents a document as a bag-of-words (BOW), much semantic context information encoded with the word order is inevitably lost. The recent surge of deep learning has drawn remarkable attention from the community. Both convolutional neural networks (CNN) [Kim, 2014] and recurrent neural network (RNN) [Mikolov et al., 2010] are widely adopted to extract semantic representation from reviews for rating prediction [Zheng et al., 2017; Catherine and Cohen, 2017; Tay et al., 2018; Wu et al., 2018; Wu et al., 2019; Li et al., 2019]. Empowered by dense representation and neural composition ability, local contextual information in reviews is well preserved and easily composited to form high-level features. These neural solutions have delivered significant performance gain over the existing BOW-based alternatives.

Many works also propose to derive aspect-level features from reviews. Several approaches utilize the external toolkit for aspect extraction [Zhang et al., 2014; He et al., 2015]. Recently, several deep learning based solutions have been proposed. These works utilize attention mechanism to extract the aspect and derive the aspect-level representations [Cheng et al., 2018; Chin et al., 2018]. By highlighting the important words or aspect, the above mentioned solutions enable better rating prediction and facilitate semantic explainability. However, the aforementioned techniques are mainly devised for the task of rating prediction. Neither of these works focuses on enhancing sequential recommendation with user reviews.

2.2 Sequential Recommendation

Sequential recommendation can be considered as a special kind of CF with implicit feedback. The effective learning of sequential patterns is widely verified to be a critical issue for sequential recommendation [Liu et al., 2009; Rendle et al., 2010; Wang et al., 2015; Hidasi et al., 2015; He and McAuley, 2016; Yu et al., 2016; Tang and Wang, 2018; Chen et al., 2018; Kang and McAuley, 2018]. Earlier methods aim to extract sequential rules based on statistic co-occurrence pattern [Agrawal and Srikant, 1995; Liu et al., 2009]. FPMC [Rendle et al., 2010] is the first work that automatically encodes the sequential relations for items with latent vectors. Afterwards, several variants are proposed to incorporate auxiliary information. Fossil integrates high-order Markov chains and item similarity together for next item recommendation [He and McAuley, 2016]. HRM proposes a hierarchical architecture to capture both sequential patterns and users' general preference [Wang et al., 2015]. Recent efforts focus on enhancing sequential recommendation by using deep learning techniques, leading to the state-of-theart performance. GRU4Rec and DREAM utilize a RNN network to capture sequential patterns [Hidasi et al., 2015; Yu et al., 2016]. RUM utilizes the attention and memory mechanisms to express and update the users' preference in a dynamic fashion [Chen et al., 2018]. The dynamic preference is modeled in both item and feature levels. Caser proposes to utilize both horizontal and vertical convolutional operations to harness the union-level and point-level sequential patterns [Tang and Wang, 2018]. They also propose a skip behavior to consider the items in the next few actions for training. SASRec builds a self-attention model for sequential recommendation [Kang and McAuley, 2018]. It can adaptively put different weights to historical items at each time step. Though these deep learning based solutions significantly enhance the performance of sequential recommendation. However, semantic information covered by user reviews are mainly overlooked. The proposed RNS in this work is the first attempt to exploit textual reviews in this field.

3 The Proposed Algorithm

In this section, we firstly give a formal formulation of reviewdriven sequential recommendation problem and then present the details of our proposed RNS.

3.1 ProblemFormulation

Let U and I represent the user and item set, respectively. Each user u ∈ U is associated with a sequence of interacted items arranged chronologically as: S u = (S u 1 , S u 2 , ..., S u |S u | ), where S u t ∈ I denotes the item purchased by user u at time step t. In this work, we exploit user reviews to enhance sequential recommendation with implicit feedback. Consequently, we merge the set of reviews written by user u and the set of reviews written for item i to form user document Du and item document Di respectively. Given user u, her recently purchased L items and their corresponding review documents, the goal is to rank candidate items in terms of the likelihood that the user will purchase in the next.

3.2 The Architecture of RNS

Figure 1 demonstrates the network architecture of RNS. The objective of RNS is to derive the item representation, the user's intrinsic preference and temporal preference at unionlevel and individual-level from the corresponding reviews. Specifically, for each user u, we learn a long-term user representation plu by utilizing an aspect-aware convolutional network (A-CNN) over user document Du. Likewise, each item i is encoded as a representation vector qi via another parallel A-CNN over item document Di.

To capture user's temporal preference denoted by psu , we utilize a hierarchical attention over attention mechanism to exploit the recent L items (i.e., qu1 , ..., quL) she has purchased at both union-level and individual-level. Then, the user's current preference is formed through a linear fusion:

where α controls the importance of the temporal preference. The preference score (i.e., purchasing likelihood) of a candidate item j is computed:

Here, we choose sigmoid function σ(x) = 1/(1 + e-x ) to meet the binary constraint (i.e., 0/1).

3.3 Aspect-Aware CNN (A-CNN)

Since the A-CNN architectures for user and item documents are identical (but with different parameters), we only describe the extraction process of user documents for simplicity. We first map each word in each user document into the corresponding embedding of d dimensions. Thus, the user document is transformed to an embedding matrix Mu ∈ Rl×d , where l is the document length. However, even for the same word, the semantics or sentimental polarity could be totally different for two different aspects in the same domain. For instance, word "low" in two sentences "The price is very low" and "This computer is low resolution" convey contrary sentiments towards aspects price and resolution respectively. So all words share the same d dimensional vector across all aspects is unreasonable. Hence, we introduce aspect-specific embedding transformation matrix Ta ∈ Rd×d and multiply it with original word embedding matrices: Mau = MuTa where Mau is the embedding matrix for Du for aspect a. Therefore, we can represent Du as a tensor M∗u ∈ Rl×d×K for K aspects. We then utilize a multi-channel convolutional operation analogous to colored images for feature extraction. Specifically, A-CNN has n filters Fk ∈ Rh×d×K , 1 ≤ k ≤ n, where h is the filter height size that can have multiple values. For example, if n = 10, h = {1, 3, 5, 7, 9}, then there will be two filters for each size. The kth filter derives its feature as follows:

where 1 ≤ i ≤ l - h + 1 is the start point of sliding window in the user document, bk is the bias, ⊙ is the convolution operator and ReLU is chosen as the activation function. The user's long-term preference is encoded as follows:

The item representation qi is extracted with the same procedure.

Intuitively, the order of purchased items in user behaviors is pivotal for the sequential recommendation. Given a user u and her recent L purchased items qu1 , qu2 , ..., quL arranged in chronological order, the latest item quL is more likely to reflect her temporal preference, while qu1 has minimal impact. Accordingly, we encode this temporal order information by including position embeddings to the item representation: qum = qum + om where om is embedding for m-th position, and 1 ≤ m ≤ L. Note that the sinusoid version of position embedding [Vaswani et al., 2017] is also investigated, but it leads to unfavorable performance, which is in line with the observation in [Kang and McAuley, 2018].

3.4 Hierarchical Attention over Attention.

We further focus on deriving the user short-term preference psu from her recent L items. A user's sequential preference could dynamically evolve over time. Moreover, different historical items could have different impacts on the user's next purchase decision. To accommodate these characteristics, we resort to the attention mechanism that has achieved great success in relevant works [Tay et al., 2018; Wu et al., 2019]. Specifically, for each of user u's L purchased items qum(1 ≤ m ≤ L) and each candidate item qj , we first calculate the weight of each qum via an attention mechanism:

Note that qum is a fusion of item representation and position embedding. This design enables RNS to easily incorporate temporal order for attention calculation. Then, the short-term preference of user u with respect to candidate item qj is computed as a weighted sum of the item embeddings as follows:

Here, ps1 u is user sequential preference representation at union-level. That is, ps1 u is jointly encoded by all previous L purchased items.

We can simply take ps1 u as the final short-term preference derived for user u, but it only captures sequential pattern at union-level, i.e., every item of past L ones has contribution for next item prediction. However, a user would buy an Apple earphone just because he bought an iPhone recently. In this case, other purchases except iPhone are just noises for making recommendation. Hence, we further explore the previous purchase records' influence on future action at individual level, i.e., identify the most relevant item. Inspired by the pointer mechanism [Tay et al., 2018], we choose the item with maximum attention weight:

where ps2 u is user sequential pattern representation at individual level. Then, we further utilize attention mechanism to discriminate which short-term preference is more important: union-level or individual-level. The final short-term user preference is calculated as follows:

The preference score is then calculated according to Equation 1-2.

3.5 Model Inference and Optimization

The model parameters of RNS include word embeddings, position embeddings, aspect transformation matrices, and two A-CNN networks. For each user u, we extract each L successive items and their next item as the target from the user's sequence S u to form a training instance. Following previous work [Tang and Wang, 2018], for each training instance tj with target item j, we randomly sample x negative items, denoted as N(j). Let C u be the collection of user u's all training instances. The objective function is defined as binary cross-entropy loss with L2 norm regularization:

where λ is coefficient for the regularization and Θ denotes all model parameters. The model is trained via Adma optimizer.

4 Experiments

In this section, we conduct experiments on three real-world datasets to evaluate our proposed RNS against uptodate stateof-the-art methods. We then perform the ablation study to investigate the validity of each design choice made in RNS. Finally, we demonstrate the superiority of the proposed hierarchical attention over attention mechanism with case study.

4.1 Experimental Setup

Datasets. We perform our experiments on the Amazon dataset 1 . This dataset contains product purchase history from Amazon ranging from May 1996 to July 2014. We conduct experiments on three categories: Instant Video (IV), Pet Supplies (PS) and Tools and Home Improvement (THI). Following prevous works [Chen et al., 2018], the users that have purchased less than 10 items are removed. We then take explicit ratings as purchase behavior. We hold the first 70% of items in each user's sequence as the training set and the rest are used for testing. After preprocessing, basic statistics of the datasets are listed in Table 1.

1 http://jmcauley.ucsd.edu/data/amazon/

Metrics. Following previous works [Kang and McAuley, 2018], for each test item, we randomly sample 100 negative items, and rank these items with the items in the testing set together. We adopt 4 widely used metrics for performance evaluation: Precision@N, Recall@N, NDCG@N and HR@N, where N indicates top-N ranked items. The first two metrics regard the recommendation task as classification problem and evaluate the Top-N classification results. NDCG@N (Normalized Discounted Cumulative Gain) evaluates ranking quality by taking the positions of ground-truth items in the ranking list into consideration. HR@N (Hit Ratio) gives the ratio of testing instances that can obtain at least one correct recommendation in the top-N.

Baseline Methods. We compare RNS with the following state-of-the-art baselines including sequential learning based models and neural network based models: (1) the popularity based recommender, Pop; (2) the factor based sequential pattern learning, FPMC [Rendle et al., 2010]; (3) RNN based sequential recommnder, GRU4Rec [Hidasi et al., 2015]; (4) attention and memory network based sequential recommnder, item-level RUM(I) and feature-level RUM(F) [Chen et al., 2018]; (5) CNN based sequential recommder, Caser [Tang andWang, 2018]; (6) self-attention based sequential recommnder, SASRec [Kang and McAuley, 2018].

Parameter Settings. As to the baselines, we utilize the recommended setting by their original work. For the proposed RNS, we set L = 5 which is a common setting in most relevant works. The number of aspects is set to be 52 , and embedding size is set to 25, n = 10 and h = {1, 3, 5, 7, 9}. The learning rate is set to 0.001, x = 3, α = 0.1 and λ = 0.0001. We set N = 5 for performance evaluation.

4.2 Results and Discussion

The overall performance of different methods is presented in Table 2. Several observations can be made:

First, the non-personalized Pop method gives the worst performance on all metrics across three datasets, which is followed by RNN based solution GRU4Rec. Also, The nonneural method FPMC surpasses GRU4Rec significantly. Here FPMC models user preference through integrating matrix factorization (MF) with sequential information, while GRU4Rec utilizes RNNs based on the hypothesis that the latter state depends on the previous one. This result might highlight that using RNN along is not sufficient to learn users' dynamic preference and long-term user preference modeled by MF in FPMC is also beneficial.

2 We observe that K is optimal at 5 on all datasets. Detailed results are not included due to page limitation.

Second, RUM(I) and RUM(F) achieve comparable results on three datasets, indicating the memory mechanism and updating operation in RUM can provide expressive ability to model user dynamic preference. Caser and SASRec outperform the other baselines on all settings except Precision@5 in Pet Supplies. Caser generally achieves better recommendation performance than other baselines in both Instant Video and Pet Supplies. In contrast, SASRec domaintes in Tools and Home Improvement.

Third, our proposed RNS consistently outperforms all baselines by a wide margin across all datasets. Specifically, the average performance gains of all metrics on three datasets are 12.8%, 10.2%, 7.9% in Instant Video, Pet Supplies and Tools and Home Improvement respectively. The substantial improvement of our model over the baselines could be credited to the following reasons: (1) we model user's long-term preference through aspect-aware convolutional processing of the review document. The rich semantics provided by the reviews help us better characterize user's general preference; (2) we capture sequential patterns at both union-level and individual-level based on a novel hierarchical attention over attention mechanism. These traits are confirmed by further analysis of RNS in the following.

4.3 Ablation Study

As there are many components in RNS architecture, we evaluate their effects via ablation study, as shown in Table 3.

Due to space limitation, we only list the results with respect to Recall and HR. Similar performance patterns are also observed for Precision and NDCG. RNS-u means that the shortterm user preference is modeled by excluding union-level, i.e., psu = ps2 u , and RNS-i for excluding individual level, i.e., psu = ps1 u . RNS-pe means that position embeddings are excluded, while RNS-at means that only one aspect is considered, i.e., without applying aspect transformation.

From Table 3, we can observe that keeping only unionlevel or individual-level preference will reduce the recommendation accuracy, suggesting that the hierarchical attention mechanism is beneficial to identify the user's dynamic preference. On both Instant Video and Pet Supplies datasets, RNS-u outperforms RNS-i on two metrics, while on Tools and Home Improvement dataset RNS-i is better. This observation suggests that the importance of union-level and individual-level depends on dataset. Without the position embeddings, RNSpe achieves worse performance than RNS on three datasets. This suggests that temporal information in forms of purchase order is a useful signal for sequential recommendation. Also, without utilizing aspect-aware feature extraction, RNS experiences the performance degradation to some extent. This suggests that aspect-aware features extracted from reviews are more discriminative.

4.4 Parameter Sensitivity

We further study the effects of parameters L and α on model performance. The results of Precision@5 on three datasets are shown. We omit the similar patterns observed for other metrics due to space limitation. Figure 2(a) plots the performance pattern by varying L values. We observe that RNS consistently achieves better performance at L = 5 across datasets. When L becomes smaller or larger, RNS experiences performance degradation to some extent. This is reasonable since a small L produces less useful historical signals and a large L would inevitably introduce much noise. Hence, we set L = 5 in our experiments.

Parameter α in Equation 1 controls the importance of sequential patterns (i.e., union-level and individual-level preferences) in RNS. Figure 2(b) plots the performance pattern by varying α values. When α = 0, RNS degrades to a variant of DeepCoNN proposed for rating prediction [Zheng et al., 2017]. We observe that RNS performs much worse in this setting. This is consistent with the existing finding that sequential patterns are crucial for sequential recommendation [Rendle et al., 2010]. The optimal performance is consistently achieved when α = 0.1 in all the three datasets. When

α becomes increasingly larger, the performance of RNS further deteriorates to a larger extent. These results further suggest that both short-term and long-term user preferences are complementary to each other for sequential recommendation. Accordingly, we set α = 0.1 in the experiments.

4.5 Case Study

To illustrate the intuition of the hierarchical attention over attention at both union-level and individual-level, We present the case study about two users sampled in Tools and Home Improvement. The attention weights in both union-level and individual-level are displayed for interpretation.

Figure 3(a) demonstrates a case where individual level is more crucial than union-level, as the user wants to buy an air conditioner cover just because he bought an air conditioner not long ago. Figure 3(b) demonstrates a completely opposite example, where all recently purchased 5 items contribute nearly equal for next purchase decision. The user needs wallpaper adhesive since he bought several kinds of wallpapers and related tools before. These observations confirm that the proposed hierarchical attention over attention mechanism in RNS is effective to capture users' dynamic preference.

5 Conclusion

In this paper, we propose a novel neural model for sequential recommendation with reviews, named RNS. Our model incorporates both a user's long-term intrinsic preference and short-term preference to predict her next actions. It utilizes aspect-aware CNN network to extract user and item representations from the corresponding review documents. A novel hierarchical attention over attention mechanism is proposed to capture sequential patterns at both union-level and individual-level. Experimental results show that RNS significantly outperforms strong state-of-the-art methods under different metrics. We will extend RNS to facilitate fine-grained textual explanation for sequential recommendation.

Acknowledgements

This research was supported by National Natural Science Foundation of China (No. 61872278, No. U1636219, No. U1804263), National Key R&D Program of China (No. 2016YFB0801303, No. 2016QY01W0105), Science and Technology Innovation Talent Project of Henan Province (No. 184200510018).