User-Based Opinion-based Recommendation

1 Introduction

Intelligent web-based systems have, for many years, taken advantage of AI techniques in a variety of online settings. Recommender systems are a case in point, bringing together ideas from knowledge representation, machine learning, and intelligent user interfaces to help match the right user with the right information at the right time.

In this paper, we explore the benefits of a recent source of recommendation knowledge - the opinions of users expressed in user generated reviews - for the purpose of recommendation. Historically, different types of recommendation techniques rely on different types of information and different sources of data. For example, collaborative filtering [Schafer et al., 2007; Resnick et al., 1994; Sarwar et al., 2001; Koren et al., 2009], rely on product ratings. whereas content and case-based techniques [Pazzani and Billsus, 2007; Bridge et al., 2005; Smyth, 2007] rely on product meta-data and features (type, price, size etc). However, product data can be hard to come by while rating data is notoriously sparse. This has led researchers to develop hybrid techniques [Burke, 2002] and approaches to harness alternative auxiliary information [Wang and Li, 2015].

The recent proliferation of user generated reviews serve as a new source of recommendation information. Reviews contain important product knowledge and valuable customer insights for use in recommendation [Pero and Horv´ath, 2013; Ganu et al., 2009]. But reviews are unstructured, and noisy, and they can even be biased. Nevertheless, opinion mining and natural language processing techniques [Liu, 2012] are now sufficiently mature for such content to act as an alternative (or complementary) source of recommendation knowledge [Chen et al., 2015]. User-generated reviews are plentiful and they contain rich product feature information including sentiment information (e.g. The desserts were beautiful, ..., but the wine was over-priced).This makes it possible to generate recommendations that are similar to those a user has liked in the past, but also better based on features that matter to the user; see [Dong and Smyth, 2016; Dong et al., 2014; Aciar et al., 2007]. Musat et al. [Musat et al., 2013] built a user interest profile for each user based on the topics mentioned in their reviews and used these profiles to produce personalized product rankings. Liu et al. [Liu et al., 2013] built user preferences based on the assumption that a user may have a higher requirement for a feature if she frequently gives a lower score for the feature compared to other users; see also the work of [Wang and Chen, 2015].

∗This work is supported by Science Foundation Ireland through the Insight Centre for Data Analytics under grant number SFI/12/RC/2289.

In this work, we consider two classical recommendation scenarios in a hotel recommendation setting. The work is an abridged version of a more detailed recent article by [Dong and Smyth, 2016].

2 Mining Profiles &Products

Extending the work of [Dong et al., 2013; 2014; 2015] we create user profiles and item/product descriptions from reviews as summarized below.

2.1 Extracting Features & Sentiment

The feature mining and sentiment analysis approaches used in this work are based on the techniques described by [Dong et al., 2014]. We mine bi-gram (adjective-noun) and singlenoun features using shallow NLP methods. In the present work we also consider additional tri-gram features, as described in the work of [Justeson and Katz, 1995], to allow us to find features like member of staff, bottle of water.

For each extracted (basic) feature, fi, from some sentence sj in review rk, we evaluate its sentiment by finding the closest sentiment word wmin to fi in sj using a sentiment lexicon [Hu and Liu, 2004]. Thus for each review we generate a set of features and their sentiment labels.

Figure 1: Examples for Clustered Features.

2.2 Feature Clusters

In reviews, people may refer to the same types of things in a variety of ways. For example, in hotel reviews, some reviewers will comment on the elevators while others will talk about lifts. This tends to produce a proliferation of features. Rather than treating these as separate features, it is more natural to recognize that they are referring to the same aspect of a hotel.

To do this we apply clustering techniques to group related features based on their similarities. Firstly, we associate each basic feature with a term-vector made up of the set of words extracted from the sentences that refer to this feature; these words are converted to lowercase, stemmed, and stop words are removed. Thus, each feature fi is associated with a set of terms and the value of each term is a normalized term frequency weight. Next, we apply a standard clustering algorithm empirically setting the target number of clusters to be 35% of the total number of features; we used CLUTO 1 . The result is a set of feature clusters such as the examples in Figure 1. These clusters act as high-level features and they are used as the basis for generating hotel and user cases as we shall describe in the next sections.

2.3 Generating Item/Product Cases

Each item/hotel (hi) is associated with a set of reviews rs(hi) = {r1, . . . , rn} and the above processes extract a set of features and clusters, c1, . . . , cm, from these reviews. Each cluster is comprised of a set of basic features and acts as a type of abstract feature, a clustered feature, with the basic features it contains related in some way. For example, in Figure 1 we can see that the features in clusters 2, and 3 are related to food as dinner, lunch respectively. Effectively each clustered feature is labelled with the cluster id, cj, and it is assigned an importance score and a sentiment score as per Equations 2 and 3. Then a hotel/item case is made up of the clustered features associated with its reviews and their corresponding importance and sentiment scores; see Equation 1. Note that cj ∈ rs(hi) means that any of the basic features in cj are present in rs(hi).

The importance score of cj , i(cj , hi), is the relative number of times that cj (or rather its basic features) is mentioned in the reviews of hotel hi.

1 http://glaros.dtc.umn.edu/gkhome/views/cluto

The sentiment score of cj, s(cj, hi), is the degree to which cj (that is, its basic features) is mentioned positively or negatively in rs(hi). Note, pos(fk, hi) and neg(fk, hi) denote the number of mentions of the basic feature fk labeled as positive or negative during the sentiment analysis phase.

2.4 Generating User Profiles

Just as we generate hotel cases from the reviews written about a specific hotel we can generate user profile cases using the reviews written by a specific user, uq as in Equation 4.

3 Ranking &Recommendation

In this section, we introduce how these user and product profiles can be used in various recommendation tasks by combining sentiment and similarity in both query-based and userbased recommendation scenarios.

3.1 Query-Based Recommendation

To begin we implement a standard non-personalized ranking approach, similar to [Dong et al., 2014], albeit based on clustered features. Consider a user uq with a hotel in mind, hq. Perhaps a hotel she has stayed in previously and she wants something similar in a new city. Let hq be the query and compare it to candidate items hc, computing the similarity and sentiment values to score each hc for ranking and recommendation to uq. For the purpose of similarity assessment we use a standard cosine metric [Pazzani and Billsus, 2007]l; see Equation 5 and note that we use the importance scores of shared features as the feature values.

Next, we calculate the sentiment score for hc. Sentiment information makes it feasible to consider not only how similar hc is to hq but also whether it enjoys better sentiment; then, we can recommend items that are not only similar to hq but have also been more positively reviewed. We do this based on a feature-by-feature sentiment comparison as per Equation 6. We can say that ci is better in hc than hq (better(ci, hq, hc) > 0) if ci in hc has a higher sentiment score than it does in hq.

We calculate the sentiment score, Sent(hq, hc) from the sum of these better scores for the features that are common to hq and hc as per Equation 7; we use C(i) to denote the clustered features of item i.

Accordingly, we can implement a non-personalized scoring function based on the above by combining Simh and Sent as per Equation 8.

3.2 User-Based Recommendation

Rather than using a specific item (hq) as a query to trigger recommendations, another common recommendation use-case is to use the user profile uq as a query. To do this we need to implement a personalized version of the approach above, by introducing user preference information into both the similarity and the sentiment calculations.

Regarding similarity, we implement a version in which we use the importance weights from the query user uq instead of the weights from hq during similarity assessment as per Equation 9. In this way, features that are more important to uq and hc play a greater role in the similarity computation.

Regarding sentiment, we propose Equation 10, which calculates a sentiment score based on the average sentiment of all hotels visited by uq with hc. We use H(uq) to denote the hotels visited by uq. Thus, the sentiment score of hc is influenced by uq's history.

For now, this means we can implement Equation 11 as the recommendation scoring function for generating personalized recommendations.

4 Evaluation

The dataset used in this work is based on the TripAdvisor dataset [Dong et al., 2014]. This dataset covers 148,575 users for 1,701 hotels. We collected the member pages of these users from TripAdvisor website, we found that these users totally have written 1,008,585 hotel reviews until July, 2014; approximately 7 reviews per user, although some users have authored significantly more. For the purpose of this work, we focus on a subset of 1,000 users with at least 5 hotel reviews. This provides a test dataset of 11,993 reviews for 10,162 hotels. Finally, for each of these hotels, we collected up to its top 100 reviews if it has for a total of 867,644 reviews.

For each of these users and hotels, we apply opinion mining to generate our feature-based descriptions. On average our test users have written 12 reviews resulting in profiles containing an average of 91 different clustered features. Likewise, the hotels are associated with an average of 89 reviews resulting in 189 clustered features per hotel on average. We will evaluate the query-based and user-based recommendation by using a standard leave-one-out style approach.

4.1 Evaluating Query-Based Recommendation

To evaluate the non-personalized, query-based recommendation strategy we use a set of test triples of the form (uq, hq, ht) corresponding to a query user uq, a query hotel, hq, and a target hotel, ht. hq and ht are in the uq's profile, but in different cities to simulate the user looking for a hotel in some new city but based on a familiar hotel. For the purpose of this test, we choose ht from a set of 8 cities in our dataset which have sufficient candidate hotels that are also in our dataset (> 80). In each triple, ht is chosen only if it has been rated as 5-star by uq; we assume the user is looking for a hotel they will like.

We note the user's rating for each query hotel in our test triples and distinguish between those with a low (2 or 3-star) rating and those that have higher (4 or 5-star) ratings; we call these 23-queries and 45-queries, respectively. This allows us to compare the ranking based on how well the query user liked the query hotel; we might expect it to be easier to identify ht from a 45-query than a 23-query. This provides us with 888 test triples; 705 have 45-queries and the remaining 183 have 23-queries. Each triple is a recommendation test, the objective of which is to locate ht based on a ranking of hotels (from the same city as ht) using hq as a query.

The results are presented in Figure 2(a) as a graph of the top-20 hit-rate - the percentage of times the target hotel is within the top-20 recommendations - versus w. At w = 0 (pure similarity-based scoring) we can see that the hit-rate is 0.20 for the 23-queries and 0.27 for the 45-queries. As expected, we can more successfully recommend the target hotel when using a 45-query than a 23-query. As we increase w (adding sentiment) the hit-rate of the query-based algorithm improves for both 45- and 23-query groups, for w up to 0.60.8; hit-rate increases from 27% (45-queries) to 35%, a relative increase of 30% compared to the similarity-only setting at w = 0. Indeed, the improvement is even more striking for the more challenging 23-queries: we see a relative improvement of 75% as hit-rate climbs from 20% to as high as 35% (at w = 0.8). There is a point after which more sentiment causes a disimprovement in hit-rate as the influence of similarity is no longer felt and sentiment tends to dominate. This point is w = 0.6 for 45-queries (and, on average, for all queries). It occurs later (w = 0.8) for the 23-queries.

We have been looking for a single hotel but the results ignore the quality of the other recommendations. For example, what is the average rating of other hotels in the top-20 recommended? And how does this change with w? This tells us about the overall quality of the recommendations and is shown in Figure 2(b) as the average TripAdvisor rating for the top-20 recommendations for increasing w. We see there is a benefit to introducing sentiment: as we increase w the av-

(a) Top 20 Hit-Rate.

erage rating of the recommended hotels increases from about 4-stars to over 4.5-stars. We can see that the average rating for the 'easier' 45-queries is higher than the average rating for the 23-queries but there is no turning point on this average rating graph because we are guiding recommendation towards hotels with more and more positive features and so we can expect their average ratings to increase accordingly.

4.2 Evaluating User-Based Recommendation

We follow a similar approach to evaluate the personalized, user-based strategy. This time we use a set of user-item test pairs, each containing a user profile uq, as a query, and a target item ht. In each case, ht is a hotel that uq has previously rated as 5-stars. Our recommendation test will use uq as a query to recommend ht. There are 665 of these test pairs. We divide profiles into small, medium, and large based on their number of reviews. small profiles have up to 10 reviews and there are 274 pairs from these with 249 unique hotels and 211 unique users; medium profiles between 11 and 20 reviews, and there are 238 pairs from these involving 215 hotels and 166 users; large profiles have more than 20 reviews and there are 153 pairs with 142 hotels and 84 users. The intuition is that small profiles will represent a tougher user-based recommendation test than medium or large profiles.

Each test pair defines a recommendation test in which uq is used to rank and recommend hotels from the same city as ht using ScoreUB. And in this test, for each test pair, we re-

(a) Top 20 Hit-Rate.

generate uq without reviews from target hotel. We calculate the top-20 hit-rate for different values of w, and the results are presented in Figure 3 (a). As before we can see how increasing w tends to improve hit-rate. For example, at w = 0 the hit-rate for large profiles is about 30% and this grows to 40% (at w = 0.4), a relative improvement of about 33%. A similar effect is noted for medium and small profiles but, as expected, the size of the effect is reduced. There is an optimal w in the range of w = 0.4 - 0.5, that seems to deliver an optimum hit-rate. Beyond this, as sentiment dominates, the hit-rate falls sharply, eventually dropping below the hit-rate achieved at w = 0 (similarity only).

For completeness, we also show the average TripAdvisor rating for the full set of 20 recommendations as w varies. Once again we see a gradual increase in recommendation quality for increasing w; the average rating of recommendation lists increases from 4-stars to about 4.6-stars.

5 Conclusions

This work (based on [Dong and Smyth, 2016]) describes and evaluates a recommendation approach to generate recommendations based on preferences mined from product reviews using a combination of shall NLP and statistical data-mining techniques. Using TripAdvisor data we show the benefits of mixing sentiment and similarity during recommendation in both query-based and user-based recommendation scenarios.