Optimal Greedy Diversity for Recommendation

1 Introduction

Recommender systems are widely used in social networks, entertainment, and eCommerce [Ricci et al., 2011]. Recommenders typically score items according to their match to the user's preferences and interests, and then recommend a list of top-scoring items. A naive selection of top items may yield a suboptimal recommendation list. For instance, collaborative filtering may recommend popular items that are known to the user [Koren and Bell, 2011]. Likewise, content-based filtering may target user's favorite topics and produce recommendations that overlook other topics [Lops et al., 2011].

This has brought to the fore the problem of diversity in recommender systems, which can be addressed by constructing recommendation lists that cover a range of user interests [Castells et al., 2011; Halvey et al., 2009; McNee et al., 2006; Vargas and Castells, 2011; Ziegler et al., 2005]. The problem is particularly acute for users with eclectic interests, the recommendations for whom should include a variety of items, to increase the chance of answering ephemeral user needs. Repercussions of the diversity problemmanifest in other recommendation use cases. Consider recommendations to heterogeneous user groups or sequential music recommendations. In both cases, the recommendation list should incorporate diverse items that either appeal to a number of group members or represent a number of music genres.

In all of the above use cases, it is important to maintain the trade-off between the diversity and utility of the results [Carbonell and Goldstein, 1998; Zhou et al., 2010]. One common solution is to strike the balance between the two objectives by maximizing a weighted sum of a modular utility function and a submodular diversity function [Qin and Zhu, 2013; Santos et al., 2010]. Another common solution is to introduce a submodular objective function that accounts for the diversity based on the utility of the recommended items in individual topics [Agrawal et al., 2009]. In both cases, the optimized function is submodular. Therefore, a (1 - 1/e)- approximate solution to the problems can be computed greedily [Nemhauser et al., 1978]. Despite being computationally efficient, the solution is suboptimal and it is well-known that the optimal solution is NP-hard to compute.

In our work, we propose a new approach to recommending diverse items, which we call diversity-weighted utility maximization (DUM). The intuition behind this method is to maximize the utility of the items recommended to users with respect to the diversity of their tastes. In other words, the utility of items remains the primary concern, but it is subjected to increasing the diversity of the recommendation list. We cast this problem as maximizing a modular utility function on the polytope of a submodular diversity function. The key difference from existing work on diversification and submodularity is that we do not maximize a submodular function; we maximize a modular function subject to a submodular constraint. This problem can be solved optimally by a greedy method [Edmonds, 1970].

We conduct an extensive evaluation of the proposed approach. We present an offline evaluation that compares DUM to three baseline methods, in terms of the trade-off between diversity and utility. We also present two online studies that compare the lists generated by DUM to baselines maximizing a convex combination of utility and diversity. All experiments show the superiority of the DUM lists over the baseline methods. Overall, we demonstrate that DUM can deliver recommendations with high degree of utility and diversity, while not requiring a-priori parameter tuning. Hence, the contribution of this work is two-fold. First, we propose a parameter-free, efficient method based on a new objective function that improves the diversity of the recommendation lists, while maintaining

Shlomo Berkovsky

CSIRO, Australia shlomo.berkovsky@csiro.au their utility. Second, we present a solid empirical evidence supporting the validity of the proposed approach.

Notation: Let A and B be sets, and e be an element of a set. We use A + e for A ∪ {e}, A + B for A ∪ B, A - e for A \ {e}, and A - B for A \ B. We represent ordered sets by vectors and also refer to them as lists.

2 RelatedWork

A common approximation to diversified ranking is based on the notion of maximal marginal relevance (MMR) [Carbonell and Goldstein, 1998]. There, utility (relevance) and diversity are represented by independent metrics. Marginal relevance is defined as a convex combination of the two metrics. Let E be the ground set of L recommendable items and w(e) be the utility of item e ∈ E. As illustrated in Algorithm 1, MMR creates a diversified ranking of the items in E by choosing an item e∗ ∈ E - S in each iteration such that it maximizes the marginal relevance:

where S is the list of recommended items, f : 2E → R+ is the diversity function, and the parameter λ controls the tradeoff between utility and diversity. Typically, the utility w is a modular function of S, whereas the diversity f is a submodular function of S, and f(S+e)-f(S) is the gain in diversity after e is added to S.

Algorithm 1 MMR: Maximal Marginal Relevance

Input: Ground set of items E S ← (), L = |E| while |S| < L do e∗ ← arg max e∈E-S (1 - λ)w(e) + λ(f (S + e) - f (S)) Append item e∗ to list S Output: List of recommended items S

Implicit approaches assume that items covering similar topics should be penalized. For instance, [Yu et al., 2014] computes f(S + e) - f(S) = -maxe′∈S sim(e, e′ ) to account for the redundancy of user intent e with respect to a set of intents S. Similarly, [Gollapudi and Sharma, 2009] targets diversification using distance functions, which are based on implicit metrics of pairwise similarity between documents. On the other hand, explicit approaches model the topics, and promote diversity by maximizing the coverage with respect to these topics. For instance, [Santos et al., 2010] defines f(S + e) - f(S) = Σt∈Tq P(t|q)P(e, ¯ S|t), where P(e, ¯ S|t) is the probability of e satisfying topic t while the ones in S failed to do so, and P(t|q) is the popularity of t among all possible topics Tq that may satisfy a query q.

Another group of related works learns a diverse ranking by maximizing a submodular objective function. Among these, [Radlinski et al., 2008] and [Yue and Guestrin, 2011] propose online learning algorithms for optimizing submodular objective functions for diversified retrieval and recommendation, respectively. The work by [Agrawal et al., 2009] addresses search diversification in an offline setting, and targets the maximization of a submodular objective function following the definition of marginal relevance. They approximate the objective function and show that an optimal solution is found when each document belongs to exactly one topic. [Vallet and Castells, 2012] studies personalization in combination with diversity, such that the objectives complement each other and satisfy user needs derived from the available user preferences.

One of the initial works in recommendation diversification is by [Ziegler et al., 2005] that proposes a metric computing the average pairwise similarity of items in a recommendation list. This metric is used to control the balance between the accuracy and diversity. [Zhang and Hurley, 2008] formulate the diversification problem as finding the best subset of items to be recommended. They address this as the maximization of the diversity of a recommendation list, subject to maintaining the accuracy of the items. [Zhou et al., 2010] proposes a hybrid method that maximizes a weighted combination of utility- and diversity-based approaches but requires parameter tuning to control the tradeoff between the two.

Most of the existing diversification approaches are based on the marginal relevance, maximizing the submodular objective function (1). Thus, a (1 - 1/e)-approximation to the optimal solution can be computed greedily [Nemhauser et al., 1978] while the exact solution to the problem is computationally intractable. The current paper is an extension to prior work in [Ashkan et al., 2014], where we introduce a new objective function for diversification, the optimal solution of which can be found greedily. This function targets the utility as the primary concern, and maximizes it with respect to the diversity of user's tastes. In this paper, we show that this method is computationally efficient and parameter-free, and it guarantees that high-utility items appear at the top of the recommendation list, as long as they contribute to the diversity of the list. We elaborate on the details of the greedy algorithm, and provide extensive online and offline evaluations on its performance.

3 Motivating Examples

Our formulation of recommending diverse items is motivated by the following problem. Suppose that a user wants to watch a movie from genre t, which is chosen randomly from a set of movie genres T . The recommender does not know t and recommends a list of movies. The user examines the list, from the first recommended item to the last, and chooses the first movie e from t. Then the user watches e and is satisfied with probability w(e). The recommender knows the satisfaction probability w(e) for each e. The goal of the recommender is to generate a list of movies that maximizes the probability that the user is satisfied for any choice of t.

We illustrate our optimization problem with two examples, where T = {t1, t2} are two movie genres. In Figure 1a, the optimal list is S = (m1,m3). If the user prefers genre t1, then user chooses m1 and is satisfied with probability 0.8. If the user prefers t2, the user chooses m3 and is satisfied with probability 0.5. Now suppose that movie m4 is replaced with m5, which satisfies the user with probability 0.9 and belongs to both genres (Figure 1b). Then the optimal list is S = (m5), as m5 is the most satisfactory movie in both genres in T .

In the next section, we state the problem of recommending diverse items slightly more formally.

4 Optimal Greedy Diversification

Let E = {1, . . . ,L} be a set of L ground items and w ∈ (R+)L be a vector of item utilities, where w(e) is the utility of item e. Let f : 2E → R+ be a diversity function, which maps any subset of E to a non-negative real number. Then the problem of diversity-weighted utility maximization is:

where A = (a1, . . . , aL) is a list of items from E, Θ is the set of all permutations of E, Ak = {a1, . . . , ak} is a set of the first k items in A, and f(Ak) - f(Ak-1) is the gain in diversity after ak is added to Ak-1. The solution to (2) is a list A∗ = (a∗1 , . . . , a∗L) that maximizes the utility of the recommended items weighted by the increase in their diversity.

The problem in Section 3 can be formulated in our framework as follows. The ground set E are all recommendable movies; the utility of movie e is the satisfaction probability of that movie; and the diversity function is defined as:

where T = {1, . . . ,M} is a finite set of topics. We state this result more formally in Proposition 1.

4.1 Greedy Solution

For a general function f, the optimization problem in (2) is NP-hard. However, when f is submodular and monotone, the problem can be cast as finding a maximum-weight basis of a polymatroid [Edmonds, 1970], which can be solved greedily and optimally. The pseudocode of our greedy algorithm is in Algorithm 2. We call it diversity-weighted utility maximization (DUM). DUM works as follows. Let A∗ = (a∗1 , . . . , a∗L) be a list of items that are ordered in decreasing order of utility, w(a∗1 ) ≥ . . . ≥ w(a∗L). DUM examines the list A∗ from the first item to the last. When f(A∗k )-f(A∗k-1 ) > 0, item a∗k is added to the recommended list S. When f(A∗k )-f(A∗k-1 ) = 0, item a∗k is not added to S because it does not contribute to the diversity of S. Finally, the algorithm returns S.

DUM has several notable properties. First, DUM does not have any tunable parameters and therefore we expect it to be robust in practice. Second, DUM is a greedy method. Therefore, it is computationally efficient. In particular, suppose that the diversity function f is an oracle that can be queried in O(1)

Input: Ground set E and item utilities w

// Compute the maximum-weight basis of a polymatroid Let a∗1 , . . . , a∗L be an ordering of items E such that: w(a∗1 ) ≥ . . . ≥ w(a∗L)

// Generate the list of recommended items S S ← ()

Output: List of recommended items S

time. Then the time complexity of DUM is O(L log L), similar to the time complexity of sorting L numbers. Finally, DUM computes the optimal solution to our problem (2).

The optimality of DUM can be proved as follows. The optimization problem (2) is equivalent to maximizing a modular function on a polymatroid [Edmonds, 1970], a well-known combinatorial optimization problem that can be solved greedily. Let M = (E, f) be a polymatroid, where E is its ground set and f is a monotone submodular function. Let:

be the independence polyhedron associated with function f. Then the maximum-weight basis of M is defined as:

where w ∈ (R+)L is a vector of non-negative weights. Since PM is a submodular polytope and w(e) ≥ 0 for all e ∈ E, the problem in (4) is equivalent to finding the order of dimensions A in which 〈w, x〉 is maximized [Edmonds, 1970]. That is, the problem can be written as (2) and has the same greedy solution. Let A∗ = (a∗1 , . . . , a∗L) be a list of items that are ordered in decreasing order of their weights, w(a∗1 ) ≥ . . . ≥ w(a∗L). Then x∗ (a∗k ) = f(A∗k )-f(A∗k-1 ) for all 1 ≤ k ≤ L.

4.2 Diversity Functions

Our approach is practical when the list of recommended items S is manageably short. Therefore, not all diversity functions f are suitable for our approach. In this section, we discuss two classes of diversity functions on topics T = {1, . . . ,M} that allow us to control the length of the recommended list.

Proposition 1. Let the diversity function f be defined as in (3). Then DUM returns a list S such that each topic t ∈ T is covered by the highest-utility item in that topic. The length of S is at most |T |.

Proof. We prove the first claim by contradiction. Let e be the highest-utility item in topic t. Suppose that item e is not chosen by DUM. Then DUM must choose another item that covers topic t. However, this contradicts to the definition of DUM. In particular, since item e is the highest-utility item in topic t, DUM must choose it before any other item in topic t. The second claim follows from the fact that f(A∗k ) - f(A∗k-1 ) > 0 implies f(A∗k ) - f(A∗k-1 ) ≥ 1. By definition, f(E) ≤ |T |. So the maximum number of items added to S is |T |.

Another suitable diversity function is:

where Nt is the number of items from topic t that is required to be in the recommended list. This function is motivated by a similar model of user behavior as in Section 3. The difference is that the user demands Nt recommended items from topic t and that the quality of the recommended list is measured by the lowest-utility item among these items, for each topic t. Under this assumption, the optimal recommended list is the list returned by DUM for the diversity function f in (5). We characterize the output of DUM for this diversity function below.

Proposition 2. Let the diversity function f be defined as in (5). Then DUM returns a list S such that each topic t ∈ T is covered by at least Nt highest-utility items in that topic. The length of S is at most ∑t∈T Nt.

When the number of topics T is huge, neither of the proposed diversity functions in (3) and (5) may be practical. In this case, we suggest reducing T by any existing dimensionality reduction method in machine learning, such as topic modeling [Blei et al., 2003]. In this particular approach, our topics T would be words, the items would be sentences, and the topics T ′ generated by topic modeling would be the new topics in (3) and (5). We leave the analysis and evaluation of this approach for future work, and focus on the case when the number of topics T is small.

5 Experiments

The proposed method is evaluated in an offline setting as well as in two online user studies. In the offline evaluation, we compare DUM to a group of existing works and under various conditions, such as recommendations across multiple users with their interest profiles defined based on different combinations of genres. In the online user studies, we choose a simpler evaluation setting in order to maintain the studies at a fair level of complexity. We choose MMR as our baseline for online studies as existing diversification approaches are mainly based on the objective function of MMR.

5.1 Offline Evaluation

We use the 1M MovieLens dataset 1 , which consists of movie ratings given on a 1-to-5 stars scale. We exclude users with less than 300 ratings, and end up with 1000 users and a total of 515k ratings.

Movies rated by each user are split randomly into the training and test set with the 2 : 1 ratio; on average, 343 movies in the training set and 171 in the test set. The split is performed three times, and the reported results are based on the average of three experiments. The training set is used for creating the user's interest profile, whereas the test set contains the recommendable movies. We use matrix factorization [Thurau et al., 2011] to predict the ratings in the test set and feed these as the utility scores into DUM and the baseline methods.

1 http://www.grouplens.org/node/12

There are 18 genres in the dataset, and each movie belongs to one genre or more. For each user, we create a multinomial distribution over the popularity of genres of the movies rated by the user in the training set, assuming that users rate movies that they watched. We sample 10 times from this distribution, to create the user's preference profile over genres, and normalize it so that the sum of the scores is 1. For each user, we set Nt = ⌊rt×K⌋ in (5), where K is the length of the recommendation list and rt is the user's preference score for genre t . That is, the coverage of a genre in the list is proportional to the degree of user preference for the genre.

Movies in the test set are used as the ground set E of recommendable movies, from which each method selects K movies. The predicted utility of the movies is used for the recommendation. The reason for using the predicted utility instead of the readily available movie ratings is to keep the evaluation close to real-world recommendation scenarios, where the utility of items is unknown. For the performance evaluation we use the actual ratings assigned by the users.

We compare DUM with three baselines. The first baseline is MMR [Carbonell and Goldstein, 1998], which is the basis of many diversification methods. The second baseline is IASelect [Agrawal et al., 2009] that targets the maximization of a submodular diversity function computed based on the utility of the recommended items for each genre. The third baseline is xQuAD [Santos et al., 2010] that considers the submodular objective function of IASelect as the diversity function and combines it linearly with the utility function in a setting similar to that of MMR. For MMR and xQuAD, we experiment with values of λ ∈ [0, 1] to account for the trade-off between diversity and utility in these methods.

The performance of DUM is compared to these baselines with respect to diversity and utility individually, as well as in combination. We use the intra-list distance (ILD) metric [Zhang and Hurley, 2008] commonly used to measure the diversity of a recommendation list as the average distance between pairs of recommended items. We compute ILD based on the Euclidean distance between the genre vectors of movies. The second metric is the discounted cumulative gain (DCG) [J¨arvelin and Kek¨al¨ainen, 2002] that measures the accumulated utility gain of items in the recommendation list, with the gain of each item discounted by its position. We compute the normalized DCG (nDCG) as nDCG = DCG/IDCG, where IDCG is the ideal gain achievable when all the items have the highest utility. The final metric is the expected intra-list distance (EILD) which was proposed by [Vargas and Castells, 2011] as a compound metric of utility and diversity. It measures the average intra-list distance with respect to rank-sensitivity and utility. We compute all the metrics for every recommendation list provided to a user. Then, we average them across the three splits, to compute user-based mean of the metric, and the mean of each metric and method is computed across all the users.

Figure 2 shows the performance of DUM and the baseline methods in terms of diversity and utility metrics for recommendation lists with K ∈ {4, 6, 8, 10}. It can be seen that both MMR and xQuAD exhibit a trade-off between the values of ILD (diversity metric) and nDCG (utility metric), for λ ∈ [0, 1]. For low values of λ the utility is prioritized, such that the diversity of the lists is low and the utility is high. An opposite is observed for high λ, when the diversity is prioritized. The performance of IASelect is the same as of xQuAD at the point of λ = 1. This is expected, as IASelect is a special case of xQuAD that accounts only for the maximization of diversity. In general, xQuAD and IASelect achieve higher utility than MMR, since in both xQuAD and IASelect the diversity function is computed based on the utility estimate of items [Santos et al., 2010].

The performance of DUM with respect to both metrics is superior to that of MMR, IASelect, and xQuAD. In particular, as shown in all four plots of Figure 2, the utility and diversity cannot be simultaneously optimized by MMR, while DUM achieves both of them, at the same time being a parameterfree method. The difference between nDCG and ILD of DUM and the corresponding values of MMR and IASelect is statistically significant at p < 0.01 (based on paired t-tests) and across the results of all users. There are small ranges of λ, where the difference between DUM and xQuAD is not significant and we further examine these ranges next.

The performance of xQuAD in terms of diversity (ILD) and utility (nDCG) for various values of λ is illustrated in Figure 3a, along with the steady performance of DUM. For ranges of λ in which the differences between DUM and xQuAD are significant at p < 0.01, the curves of xQuAD are thicker. It can be seen that λ = 0.74 is the operating point of xQuAD where the utility and the diversity curves intersect. There, xQuAD slightly outperforms DUM in utility, but this is not statistically significant. At the same point, DUM outperforms xQuAD in terms of diversity, and this observation is significant. This confirms that the utility and diversity of DUM are either on par or superior to those of xQuAD at its operating point.

Another argument in favor of DUM is obtained through the EILD metric that combines diversity and utility. A comparison of DUM and xQuAD with respect to EILD is shown in Figure 3-b. It can be seen that DUM substantially outperforms xQuAD for all the values of λ, confirming the superiority of DUM in balancing the utility and diversity goals.

5.2 Online User Studies

We conduct two online studies using Amazon's Mechanical Turk (MT) 2 . In the first study, we evaluate the lists generated by DUM and MMR, by asking MT workers to identify in the lists a movie that matches their genre of interest and to indicate the relevance of this movie. In the second study, we compare the lists generated by DUM and MMR, by asking MT workers to judge the coverage of two movie genres by the lists.

Study 1

The ground set E are 10k most frequently rated IMDb 3 movies. The utility w(e) of movie e is the number of ratings assigned to the movie. The values of w(e) are normalized such that maxe∈E w(e) = 1. The diversity function f is defined as in (3). We also normalize f such that maxe∈E f(e) = 1. The topics T are 8 most popular movie genres in our dataset, T = {Drama, Comedy, Thriller, Romance, Action, Crime, Adventure, Horror}. For this T , DUM generates a list of up to 8 movies (see Proposition 1). We compare DUM to three variants of MMR, which are parameterized by λ ∈ { 1 3 , 23 , 0.99}.

2 http://www.mturk.com

3 http://www.imdb.com

All methods are evaluated in 200 MT tasks (HITs). In each HIT, we ask the worker to choose a genre of interest. Then, we generate four recommendation lists: one by DUM and three by MMR for different values of λ. We ask the worker to evaluate each list with two questions. First, we ask the worker to identify a movie that matches the chosen genre. This question assesses whether the chosen genre is covered by the list (the worker can answer "none"). If a matching movie is identified, we ask if this movie is a good recommendation for the chosen genre. This question assesses whether the chosen genre is covered by a good movie.

In each HIT, the four recommendation lists are presented in a random order. This eliminates the position bias. Moreover, in each HIT, the recommendable items are 3.3k randomly chosen movies from the ground set E of 10k movies. Thus, the recommendation lists differ across the HITs, which eliminates the item bias. Finally, all the lists are of the same length - that of the list produced by DUM. This eliminates any potential bias due to a different length of the lists.

The 200 HITs are completed by 34 master workers, who were assigned with this status based on the quality of their prior HITs. Each worker is allowed to complete at most 8 HITs. On average, each HIT is completed in 72 seconds, so each recommendation list is evaluated in 19 seconds on average. The results of the study are presented in Table 1. For each method, we report the percentage of times when the matching movie is among top K movies in the list and is considered a good recommendation. We report K ∈ {2, 4, 6, 8}. We observe that the percentage of times that the worker finds a matching movie in the DUM list is significantly higher than in the list of the best performing baseline, MMR with λ = 13 . This result is statistically significant for larger values of K.

Note that for all the methods in Table 1, the ratio between the percentage of times that the movie is a good recommendation and that the matching movie is found is between 0.92 and 0.94. This implies that if a matching movie is found, it is very likely to be a good recommendation. We conjecture that this is due to the popularity of movies in E, which practically guarantees the high utility of the movies and erodes the differences between the compared methods.

Study 2

In the second study, we evaluate DUM on a specific use case of recommending a diverse set of movies that covers exactly two genres. In each HIT, we ask the MT worker to consider a situation where Alice and Bob, who prefer two different genres, go for a vacation and can take with them several movies. We compare DUM to three variants of MMR parameterized by λ ∈ { 13 , 23 , 0.99}, and generate four lists: one by DUM and three by MMR for these values of λ. For each list, we ask the worker to indicate whether the list is appropriate for both Alice and Bob, only for one of them, or for neither. The lists are presented in a random order to eliminate the position bias.

Each HIT is associated with two genres preferred by Alice and Bob, t1 and t2. We generate three HITs for each pair of the 18 most frequent IMDb genres. So the recommendation lists are evaluated 3 × 18×17 2 = 459 times. Like in the first study, the ground set E are 10K most frequently rated IMDb movies. The utility w(e) is the number of ratings assigned to e. The diversity function f is defined as in (5). We set Nt1 = Nt2 = 4. In this case, DUM generates a list of at most 8 movies, at least 4 from each genre. The length of the other lists is set to that of DUM. The utility and diversity are normalized as in the first study.

The 459 HITs are completed by 57 master workers. Each worker is allowed to complete at most 10 HITs. On average, each HIT is completed in 57 seconds, so that each list is evaluated in 14 seconds. The results of the study are presented in Table 2. For each method, we report the percentage of times that the worker considers the recommendation list as suitable for both Alice and Bob, only for one of them, or for neither. We observe that the workers consider the DUM list to be suitable for both Alice and Bob in 74.51% of cases. This is 9.6% higher than the best performing baseline, MMR with λ = 1 3 . This difference is statistically significant at p < 0.01. Hence, DUM is perceived superior to MMR in generating diverse lists that cover exactly two movie genres.

6 Conclusion

In this work, we propose a new approach to diversifying a list of recommended items, DUM, which maximizes the utility of the items subject to the increase in their diversity. We show that the problem can be solved optimally by a greedy method, because it is an instance of maximizing a modular function on the polytope of a submodular function. We evaluate DUM on a variety of problems. In the offline experiments, we compare DUM to three popular baselines in terms of the utility and diversity of recommended lists. Our results show that DUM effectively balances the utility and diversity of the lists, despite the fact that it has no tunable parameters. We also present two online user studies and show that DUM outperforms a baseline that maximizes a linear combination of utility and diversity.

A future direction for this work is to account for the novelty of the recommended items [Castells et al., 2011; Clarke et al., 2008] with respect to prior consumption history of the user. This may be incorporated into the diversity function by considering, apart from the diversity contribution, also the novelty contribution of the items. Moreover, similar to [Radlinski et al., 2008; Yue and Guestrin, 2011], the utilities of items can be learned in an online fashion using the learning variants of maximizing a modular function on a polymatroid [Kveton et al., 2014a; 2014b]. Another issue that deserves investigation is the changes in the diversity function needed to reflect the tolerance for redundancy in different domains. For instance, a diversity metric for news filtering may differ from the metric we derived here for the movie recommendation task. We intend to address these questions in the future