{"type":"ScholarlyArticle","title":"On Sampled Metrics for Item Recommendation (Extended Abstract)","authors":[{"name":"Walid Krichene","slug":"walid-krichene","given":"Walid","family":"Krichene","affiliation":"Google Research"},{"name":"Steffen Rendle","slug":"steffen-rendle","given":"Steffen","family":"Rendle","affiliation":"Google Research"}],"abstract":"Recommender systems personalize content by recommending items to users. Item recommendation algorithms are evaluated by metrics that compare the positions of truly relevant items among the recommended items. To speed up the computation of metrics, recent work often uses sampled metrics where only a smaller set of random items and the relevant items are ranked. This paper investigates such sampled metrics and shows that they are inconsistent with their exact counterpart, in the sense that they do not persist relative statements, e.g., recommender A is better than B, not even in expectation. We show that it is possible to improve the quality of the sampled metrics by applying a correction. We conclude with an empirical evaluation of the naive sampled metrics and their corrected variants. Our work suggests that sampling should be avoided for metric calculation, however if an experimental study needs to sample, the proposed corrections can improve the estimates.","doi":"10.24963/ijcai.2021/651","url":"https://ijcai.org/proceedings/2021/651","pdf":"https://ijcai.org/proceedings/2021/0651.pdf","markdown":"https://ijcai.org/proceedings/2021/651.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/651","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":651,"pages":{"first":4784,"last":4788},"keywords":["Machine Learning: Recommender Systems"],"videos":[],"references":[{"position":1,"raw":"[Ebesu et al., 2018] Travis Ebesu, Bin Shen, and Yi Fang. Collaborative memory network for recommendation sys- tems. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SI- GIR '18, pages 515-524. ACM, 2018.","title":"Collaborative memory network for recommendation systems","authors":"Travis Ebesu, Bin Shen, Yi Fang","year":"2018","venue":"The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SI-GIR '18","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Harper and Konstan, 2015] F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4), December 2015.","title":"The movielens datasets: History and context","authors":"F, Maxwell Harper, Joseph A Konstan","year":"2015","venue":"ACM Trans. Interact. Intell. Syst","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[He et al., 2017] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collab- orative filtering. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pages 173- 182, 2017.","title":"Neural collaborative filtering","authors":"He","year":"2017","venue":"Proceedings of the 26th International Conference on World Wide Web, WWW '17","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hu et al., 2018] Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S. Yu. Leveraging meta-path based context for top-n recommendation with a neural co-attention model. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18, pages 1531-1540. ACM, 2018.","title":"Leveraging meta-path based context for top-n recommendation with a neural co-attention model","authors":"Binbin Hu, Chuan Shi, Wayne Xin Zhao, Philip S Yu","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Krichene and Rendle, 2020] Walid Krichene and Steffen Rendle. On sampled metrics for item recommendation. In Proceedings of the 26th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, KDD '20, pages 1748-1757. ACM, 2020.","title":"On sampled metrics for item recommendation","authors":"Rendle Krichene, Walid Krichene, Steffen Rendle","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Krichene et al., 2019] Walid Krichene, Nicolas Mayoraz, Steffen Rendle, Li Zhang, Xinyang Yi, Lichan Hong, Ed Chi, and John Anderson. Efficient training on very large corpora via gramian estimation. In International Conference on Learning Representations, 2019.","title":"Efficient training on very large corpora via gramian estimation","authors":"Walid Krichene, Nicolas Mayoraz, Steffen Rendle, Li Zhang, Xinyang Yi, Lichan Hong, Ed Chi, John Anderson","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Wang et al., 2019] Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, Yixin Cao, and Tat-Seng Chua. Ex- plainable reasoning over knowledge graphs for recommen- dation. In Proceedings of the 33rd AAAI Conference on Ar- tificial Intelligence (AAAI), AAAI '19, pages 5329-5336, 2019.","title":"Explainable reasoning over knowledge graphs for recommendation","authors":"Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, Yixin Cao, Tat-Seng Chua","year":"2019","venue":"Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), AAAI '19","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Yang et al., 2018a] Longqi Yang, Eugene Bagdasaryan, Joshua Gruenstein, Cheng-Kang Hsieh, and Deborah Es- trin. Openrec: A modular framework for extensible and adaptable recommendation algorithms. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18, pages 664-672. ACM, 2018.","title":"Openrec: A modular framework for extensible and adaptable recommendation algorithms","authors":"Yang","year":"2018","venue":"Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Yang et al., 2018b] Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Un- biased offline recommender evaluation for missing-not-at- random implicit feedback. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18, pages 279-287. ACM, 2018.","title":"Unbiased offline recommender evaluation for missing-not-atrandom implicit feedback","authors":"Yang","year":"2018","venue":"Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Yu et al., 2017] Hsiang-Fu Yu, Mikhail Bilenko, and Chih- Jen Lin. Selection of negative samples for one-class matrix factorization. In Proceedings of the 2017 SIAM Interna- tional Conference on Data Mining, pages 363-371, 2017.","title":"Selection of negative samples for one-class matrix factorization","authors":"Hsiang-Fu Yu, Mikhail Bilenko, Chih-Jen Lin","year":"2017","venue":"Proceedings of the 2017 SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}