{"type":"ScholarlyArticle","title":"Methodological Issues in Recommender Systems Research (Extended Abstract)","authors":[{"name":"Maurizio Ferrari Dacrema","slug":"maurizio-ferrari-dacrema","given":"Maurizio","family":"Ferrari Dacrema","affiliation":"Politecnico di Milano , Italy"},{"name":"Paolo Cremonesi","slug":"paolo-cremonesi","given":"Paolo","family":"Cremonesi","affiliation":"Politecnico di Milano , Italy"},{"name":"Dietmar Jannach","slug":"dietmar-jannach","given":"Dietmar","family":"Jannach","affiliation":"University of Klagenfurt , Austria"}],"abstract":"The development of continuously improved machine learning algorithms for personalized item ranking lies at the core of today's research in the area of recommender systems.\r\nOver the years, the research community has developed widely-agreed best practices for comparing algorithms and demonstrating progress with offline experiments. Unfortunately, we find this accepted research practice can easily lead to phantom progress due to the following reasons: limited reproducibility, comparison with complex but weak and non-optimized baseline algorithms, over-generalization from a small set of experimental configurations. To assess the extent of such problems, we analyzed 18 research papers published recently at top-ranked conferences. Only 7 were reproducible with reasonable effort, and 6 of them could often be outperformed by relatively simple heuristic methods, e.g., nearest neighbors. In this paper, we discuss these observations in detail, and reflect on the related fundamental problem of over-reliance on offline experiments in recommender systems research.","doi":"10.24963/ijcai.2020/650","url":"https://ijcai.org/proceedings/2020/650","pdf":"https://ijcai.org/proceedings/2020/0650.pdf","markdown":"https://ijcai.org/proceedings/2020/650.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/650","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":650,"pages":{"first":4706,"last":4710},"keywords":["Machine Learning: Recommender Systems","Machine Learning: Deep Learning","Multidisciplinary Topics and Applications: Validation and Verification"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25242/file_25242.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Armstrong et al., 2009] Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. Improvements that don't add up: Ad-hoc retrieval results since 1998. In CIKM '09, pages 601-610, 2009.","title":"Improvements that don't add up: Ad-hoc retrieval results since 1998","authors":"Timothy G Armstrong, Alistair Moffat, William Webber, Justin Zobel","year":"2009","venue":"CIKM '09","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cooper et al., 2014] Colin Cooper, Sang Hyuk Lee, Tomasz Radzik, and Yiannis Siantos. Random walks in recom- mender systems: exact computation and simulations. In WWW '14, pages 811-816, 2014.","title":"Random walks in recommender systems: exact computation and simulations","authors":"Colin Cooper, Sang Hyuk Lee, Tomasz Radzik, Yiannis Siantos","year":"2014","venue":"WWW '14","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cremonesi et al., 2010] Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. Performance of recommender algo- rithms on top-n recommendation tasks. In Proceedings of the 4th ACM Conference on Recommender Systems (Rec- Sys '10), pages 39-46, 2010.","title":"Performance of recommender algorithms on top-n recommendation tasks","authors":"Paolo Cremonesi, Yehuda Koren, Roberto Turrin","year":"2010","venue":"Proceedings of the 4th ACM Conference on Recommender Systems (Rec-Sys '10)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cremonesi et al., 2012] Paolo Cremonesi, Franca Garzotto, and Roberto Turrin. Investigating the persuasion potential of recommender systems from a quality perspective: An empirical study. ACM Transactions on Interactive Intelli- gent Systems (TiiS), 2(2):1-41, 2012.","title":"Investigating the persuasion potential of recommender systems from a quality perspective: An empirical study","authors":"Paolo Cremonesi, Franca Garzotto, Roberto Turrin","year":"2012","venue":"ACM Transactions on Interactive Intelligent Systems (TiiS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ebesu et al., 2018] Travis Ebesu, Bin Shen, and Yi Fang. Collaborative memory network for recommendation sys- tems. SIGIR '18, 2018.","title":"Collaborative memory network for recommendation systems","authors":"Travis Ebesu, Bin Shen, Yi Fang","year":"2018","venue":"SIGIR '18","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ferrari Dacrema et al., 2019a] Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. A troubling analysis of reproducibility and progress in rec- ommender systems research. arXiv:1911.07698, 2019.","title":"A troubling analysis of reproducibility and progress in recommender systems research","authors":"Ferrari Dacrema","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ferrari Dacrema et al., 2019b] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. RecSys '19, 2019.","title":"Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches","authors":"Ferrari Dacrema","year":"2019","venue":"RecSys '19","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gomez-Uribe and Hunt, 2015] Carlos A. Gomez-Uribe and Neil Hunt. The Netflix recommender system: Algorithms, business value, and innovation. Transactions on Manage- ment Information Systems, 6(4):13:1-13:19, 2015.","title":"The Netflix recommender system: Algorithms, business value, and innovation","authors":"Carlos A Gomez-Uribe, Neil Hunt","year":"2015","venue":"Transactions on Management Information Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2017] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collabo- rative filtering. In WWW '17, pages 173-182, 2017.","title":"Neural collaborative filtering","authors":"He","year":"2017","venue":"WWW '17","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2018] Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S Yu. Leveraging meta-path based context for top-n recommendation with a neural co-attention model. In KDD '18, pages 1531-1540, 2018.","title":"Leveraging meta-path based context for top-n recommendation with a neural co-attention model","authors":"Binbin Hu, Chuan Shi, Wayne Xin Zhao, Philip S Yu","year":"2018","venue":"KDD '18","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jannach and Jugovac, 2019] Dietmar Jannach and Michael Jugovac. Measuring the business value of recommender systems. ACM Transactions on Management Information Systems, 10(4), 2019.","title":"Measuring the business value of recommender systems","authors":"Dietmar Jannach, Michael Jugovac","year":"2019","venue":"ACM Transactions on Management Information Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jannach et al., 2012] Dietmar Jannach, Markus Zanker, Mouzhi Ge, and Marian Gröning. Recommender systems in computer science and information systems -a landscape of research. In EC-Web 2012, pages 76-87, 2012.","title":"Recommender systems in computer science and information systems -a landscape of research","authors":"Dietmar Jannach, Markus Zanker, Mouzhi Ge, Marian Gröning","year":"2012","venue":"EC-Web 2012","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jannach et al., 2016] Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. Recommender systems -beyond matrix completion. Communications of the ACM, 59(11):94-102, 2016.","title":"Recommender systems -beyond matrix completion","authors":"Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, Markus Zanker","year":"2016","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levy and Jack, 2013] Mark Levy and Kris Jack. Efficient top-n recommendation by linear regression. In RecSys Large Scale Recommender Systems Workshop, 2013.","title":"Efficient top-n recommendation by linear regression","authors":"Jack ; Levy, Mark Levy, Kris Jack","year":"2013","venue":"RecSys Large Scale Recommender Systems Workshop","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and She, 2017] Xiaopeng Li and James She. Collabora- tive variational autoencoder for recommender systems. In KDD '17, pages 305-314, 2017.","title":"Collaborative variational autoencoder for recommender systems","authors":"Xiaopeng Li, James She","year":"2017","venue":"KDD '17","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2018] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational au- toencoders for collaborative filtering. In WWW '18, pages 689-698, 2018.","title":"Variational autoencoders for collaborative filtering","authors":"Dawen Liang, G Rahul, Matthew D Krishnan, Tony Hoffman, Jebara","year":"2018","venue":"WWW '18","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin, 2019] Jimmy Lin. The neural hype and comparisons against weak baselines. SIGIR Forum, 52(2):40-51, Jan- uary 2019.","title":"The neural hype and comparisons against weak baselines","authors":"Jimmy Lin","year":"2019","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lipton and Steinhardt, 2018] Zachary C. Lipton and Jacob Steinhardt. Troubling trends in machine learning scholar- ship. arXiv:1911.07698, 2018.","title":"Troubling trends in machine learning scholarship","authors":"Zachary C Lipton, Jacob Steinhardt","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ludewig and Jannach, 2018] Malte Ludewig and Dietmar Jannach. Evaluation of session-based recommendation al- gorithms. User-Modeling and User-Adapted Interaction, 28(4-5):331-390, 2018.","title":"Evaluation of session-based recommendation algorithms","authors":"Malte Ludewig, Dietmar Jannach","year":"2018","venue":"User-Modeling and User-Adapted Interaction","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ludewig et al., 2019] Malte Ludewig, Noemi Mauro, Sara Latifi, and Dietmar Jannach. Performance comparison of neural and non-neural approaches to session-based recom- mendation. In RecSys '19, 2019.","title":"Performance comparison of neural and non-neural approaches to session-based recommendation","authors":"Malte Ludewig, Noemi Mauro, Sara Latifi, Dietmar Jannach","year":"2019","venue":"RecSys '19","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Makridakis et al., 2018] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. Statistical and machine learning forecasting methods: Concerns and ways forward. PloS one, 13(3), 2018.","title":"Statistical and machine learning forecasting methods: Concerns and ways forward","authors":"Spyros Makridakis, Evangelos Spiliotis, Vassilios Assimakopoulos","year":"2018","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Maksai et al., 2015] Andrii Maksai, Florent Garcin, and Boi Faltings. Predicting online performance of news recom- mender systems through richer evaluation metrics. In Rec- Sys '15, pages 179-186, 2015.","title":"Predicting online performance of news recommender systems through richer evaluation metrics","authors":"Andrii Maksai, Florent Garcin, Boi Faltings","year":"2015","venue":"Rec-Sys '15","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nickerson, 1998] Raymond S. Nickerson. Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2):175-220, 1998.","title":"Confirmation bias: A ubiquitous phenomenon in many guises","authors":"Raymond S Nickerson","year":"1998","venue":"Review of General Psychology","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Paudel et al., 2017] Bibek Paudel, Fabian Christoffel, Chris Newell, and Abraham Bernstein. Updatable, accurate, di- verse, and scalable recommendations for interactive appli- cations. ACM Transactions on Interactive Intelligent Sys- tems, 7(1):1, 2017.","title":"Updatable, accurate, diverse, and scalable recommendations for interactive applications","authors":"Bibek Paudel, Fabian Christoffel, Chris Newell, Abraham Bernstein","year":"2017","venue":"ACM Transactions on Interactive Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rendle et al., 2019] Steffen Rendle, Li Zhang, and Yehuda Koren. On the difficulty of evaluating baselines: A study on recommender systems. arXiv:1905.01395, 2019.","title":"On the difficulty of evaluating baselines: A study on recommender systems","authors":"Steffen Rendle, Li Zhang, Yehuda Koren","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ricci et al., 2011] Francesco Ricci, Lior Rokach, and Bracha Shapira. Introduction to recommender systems handbook. In Recommender systems handbook, pages 1- 35. Springer, 2011.","title":"Introduction to recommender systems handbook","authors":"Francesco Ricci, Lior Rokach, Bracha Shapira","year":"2011","venue":"Recommender systems handbook","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2015] Hao Wang, Naiyan Wang, and Dit-Yan Yeung. Collaborative deep learning for recommender sys- tems. In KDD '15, pages 1235-1244, 2015.","title":"Collaborative deep learning for recommender systems","authors":"Hao Wang, Naiyan Wang, Dit-Yan Yeung","year":"2015","venue":"KDD '15","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2018] Lei Zheng, Chun-Ta Lu, Fei Jiang, Ji- awei Zhang, and Philip S. Yu. Spectral collaborative filter- ing. In RecSys '18, pages 311-319, 2018.","title":"Spectral collaborative filtering","authors":"Lei Zheng, Chun-Ta Lu, Fei Jiang, Jiawei Zhang, Philip S Yu","year":"2018","venue":"RecSys '18","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}