{"type":"ScholarlyArticle","title":"Impact of Consuming Suggested Items on the Assessment of Recommendations in User Studies on Recommender Systems","authors":[{"name":"Benedikt Loepp","slug":"benedikt-loepp","given":"Benedikt","family":"Loepp","affiliation":"University of Duisburg-Essen"},{"name":"Tim Donkers","slug":"tim-donkers","given":"Tim","family":"Donkers","affiliation":"University of Duisburg-Essen"},{"name":"Timm Kleemann","slug":"timm-kleemann","given":"Timm","family":"Kleemann","affiliation":"University of Duisburg-Essen"},{"name":"Jürgen Ziegler","slug":"jurgen-ziegler","given":"Jürgen","family":"Ziegler","affiliation":"University of Duisburg-Essen"}],"abstract":"User studies are increasingly considered important in research on recommender systems. Although participants typically cannot consume any of the recommended items, they are often asked to assess the quality of recommendations and of other aspects related to user experience by means of questionnaires. Not being able to listen to recommended songs or to watch suggested movies, might however limit the validity of the obtained results. Consequently, we have investigated the effect of consuming suggested items. In two user studies conducted in different domains, we showed that consumption may lead to differences in the assessment of recommendations and in questionnaire answers. Apparently, adequately measuring user experience is in some cases not possible without allowing users to consume items. On the other hand, participants sometimes seem to approximate the actual value of recommendations reasonably well depending on domain and provided information.","doi":"10.24963/ijcai.2019/863","url":"https://ijcai.org/proceedings/2019/863","pdf":"https://ijcai.org/proceedings/2019/0863.pdf","markdown":"https://ijcai.org/proceedings/2019/863.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/863","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":863,"pages":{"first":6201,"last":6205},"keywords":["Machine Learning: Experimental Methodology;Replicability","Multidisciplinary Topics and Applications: Recommender Systems","Humans and AI: Human-Computer Interaction","Humans and AI: Personalization and User Modeling"],"videos":[],"references":[{"position":1,"raw":"[Adomavicius et al., 2013] Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang. Do recommender systems manipulate consumer prefer- ences? A study of anchoring effects. Information Systems Research, 24(4):956-975, 2013.","title":"Do recommender systems manipulate consumer preferences? A study of anchoring effects","authors":"Gediminas Adomavicius, Jesse C Bockstedt, Shawn P Curley, Jingjing Zhang","year":"2013","venue":"Information Systems Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bilgic and Mooney, 2005] Mustafa Bilgic and Raymond J. Mooney. Explaining recommendations: Satisfaction vs. promotion. In Proceedings of the Beyond Personalization Workshop, 2005.","title":"Explaining recommendations: Satisfaction vs. promotion","authors":"Mustafa Bilgic, Raymond J Mooney","year":"2005","venue":"Proceedings of the Beyond Personalization Workshop","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bollen et al., 2012] Dirk Bollen, Mark P. Graus, and Mar- tijn C. Willemsen. Remembering the stars? Effect of time on preference retrieval from memory. In RecSys '12: Proceedings of the 6th ACM Conference on Recommender Systems, pages 217-220. ACM, 2012.","title":"Remembering the stars? Effect of time on preference retrieval from memory","authors":"Dirk Bollen, Mark P Graus, Martijn C Willemsen","year":"2012","venue":"RecSys '12: Proceedings of the 6th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cosley et al., 2003] Dan Cosley, Shyong K. Lam, Istvan Al- bert, Joseph A. Konstan, and John Riedl. Is seeing believ- ing? How recommender interfaces affect users' opinions. In CHI '03: Proceedings of the 21st ACM Conference on Human Factors in Computing Systems, pages 585-592. ACM, 2003.","title":"Is seeing believing? How recommender interfaces affect users' opinions","authors":"Dan Cosley, K Shyong, Istvan Lam, Joseph A Albert, John Konstan, Riedl","year":"2003","venue":"CHI '03: Proceedings of the 21st ACM Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dijksterhuis and van Olden, 2006] Ap Dijksterhuis and Zeger van Olden. On the benefits of thinking uncon- sciously: Unconscious thought can increase post-choice satisfaction. Journal of Experimental Social Psychology, 42(5):627-631, 2006.","title":"On the benefits of thinking unconsciously: Unconscious thought can increase post-choice satisfaction","authors":"Dijksterhuis, Ap Van Olden ;, Zeger Dijksterhuis, Van Olden","year":"2006","venue":"Journal of Experimental Social Psychology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Graus and Willemsen, 2016] Mark P. Graus and Martijn C. Willemsen. Can trailers help to alleviate popularity bias in choice-based preference elicitation? In IntRS '16: Pro- ceedings of the 3rd Joint Workshop on Interfaces and Hu- man Decision Making for Recommender Systems, pages 22-27, 2016.","title":"Can trailers help to alleviate popularity bias in choice-based preference elicitation? In IntRS '16","authors":"Mark P Graus, Martijn C Willemsen","year":"2016","venue":"Proceedings of the 3rd Joint Workshop on Interfaces and Human Decision Making for Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gunawardana and Shani, 2015] Asela Gunawardana and Guy Shani. Recommender Systems Handbook, chap- ter Evaluating Recommender Systems, pages 265-308. Springer US, 2015.","title":"Recommender Systems Handbook, chapter Evaluating Recommender Systems","authors":"Asela Gunawardana, Guy Shani","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hill et al., 1995] Will Hill, Larry Stead, Mark Rosenstein, and George Furnas. Recommending and evaluating choices in a virtual community of use. In CHI '95: Pro- ceedings of the 13th ACM Conference on Human Factors in Computing Systems, pages 194-201. ACM, 1995.","title":"Recommending and evaluating choices in a virtual community of use","authors":"Hill","year":"1995","venue":"CHI '95: Proceedings of the 13th ACM Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Knijnenburg and Willemsen, 2015] Bart P. Knijnenburg and Martijn C. Willemsen. Recommender Systems Handbook, chapter Evaluating Recommender Systems with User Ex- periments, pages 309-352. Springer US, 2015.","title":"Recommender Systems Handbook, chapter Evaluating Recommender Systems with User Experiments","authors":"Bart P Knijnenburg, Martijn C Willemsen","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Knijnenburg et al., 2011] Bart P. Knijnenburg, Martijn C. Willemsen, and Alfred Kobsa. A pragmatic procedure to support the user-centric evaluation of recommender sys- tems. In RecSys '11: Proceedings of the 5th ACM Con- ference on Recommender Systems, pages 321-324. ACM, 2011.","title":"A pragmatic procedure to support the user-centric evaluation of recommender systems","authors":"Bart P Knijnenburg, C Martijn, Alfred Willemsen, Kobsa","year":"2011","venue":"RecSys '11: Proceedings of the 5th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konstan and Riedl, 2012] Joseph A. Konstan and John Riedl. Recommender systems: From algorithms to user experience. User Modeling and User-Adapted Interaction, 22(1-2):101-123, 2012.","title":"Recommender systems: From algorithms to user experience","authors":"Joseph A Konstan, John Riedl","year":"2012","venue":"User Modeling and User-Adapted Interaction","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Loepp et al., 2018] Benedikt Loepp, Tim Donkers, Timm Kleemann, and Jürgen Ziegler. Impact of item consump- tion on assessment of recommendations in user studies. In RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems, pages 49-53. ACM, 2018.","title":"Impact of item consumption on assessment of recommendations in user studies","authors":"Benedikt Loepp, Tim Donkers, Timm Kleemann, Jürgen Ziegler","year":"2018","venue":"RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McNee et al., 2006] Sean M. McNee, John Riedl, and Joseph A. Konstan. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In CHI '06: Extended Abstracts on Human Factors in Computing Systems, pages 1097-1101. ACM, 2006.","title":"Being accurate is not enough: How accuracy metrics have hurt recommender systems","authors":"Sean M Mcnee, John Riedl, Joseph A Konstan","year":"2006","venue":"CHI '06: Extended Abstracts on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nanou et al., 2010] Theodora Nanou, George Lekakos, and Konstantinos Fouskas. The effects of recommendations' presentation on persuasion and satisfaction in a movie rec- ommender system. Multimedia Systems, 16(4-5):219-230, 2010.","title":"The effects of recommendations' presentation on persuasion and satisfaction in a movie recommender system","authors":"Theodora Nanou, George Lekakos, Konstantinos Fouskas","year":"2010","venue":"Multimedia Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pu et al., 2011] Pearl Pu, Li Chen, and Rong Hu. A user- centric evaluation framework for recommender systems. In RecSys '11: Proceedings of the 5th ACM Conference on Recommender Systems, pages 157-164. ACM, 2011.","title":"A usercentric evaluation framework for recommender systems","authors":"Pu","year":"2011","venue":"RecSys '11: Proceedings of the 5th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Said and Bellogín, 2018] Alan Said and Alejandro Bel- logín. Coherence and inconsistencies in rating behavior: Estimating the magic barrier of recommender systems. User Modeling and User-Adapted Interaction, 2018.","title":"Coherence and inconsistencies in rating behavior: Estimating the magic barrier of recommender systems","authors":"Alan Said, Alejandro Bellogín","year":"2018","venue":"User Modeling and User-Adapted Interaction","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sharma and Cosley, 2013] Amit Sharma and Dan Cosley. Do social explanations work? Studying and modeling the effects of social explanations in recommender systems. In WWW '13: Proceedings of the 22nd International Confer- ence on World Wide Web, pages 1133-1144. ACM, 2013.","title":"Do social explanations work? Studying and modeling the effects of social explanations in recommender systems","authors":"Amit Sharma, Dan Cosley","year":"2013","venue":"WWW '13: Proceedings of the 22nd International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tintarev and Masthoff, 2008] Nava Tintarev and Judith Masthoff. Over-and underestimation in different prod- uct domains. In Proceedings of the ECAI Workshop on Recommender Systems, pages 14-19, 2008.","title":"Over-and underestimation in different product domains","authors":"Nava Tintarev, Judith Masthoff","year":"2008","venue":"Proceedings of the ECAI Workshop on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tintarev and Masthoff, 2012] Nava Tintarev and Judith Masthoff. Evaluating the effectiveness of explanations for recommender systems. User Modeling and User-Adapted Interaction, 22(4-5):399-439, 2012.","title":"Evaluating the effectiveness of explanations for recommender systems","authors":"Nava Tintarev, Judith Masthoff","year":"2012","venue":"User Modeling and User-Adapted Interaction","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tintarev and Masthoff, 2015] Nava Tintarev and Judith Masthoff. Recommender Systems Handbook, chapter Ex- plaining Recommendations: Design and Evaluation, pages 353-382. Springer US, 2015.","title":"Recommender Systems Handbook, chapter Explaining Recommendations: Design and Evaluation","authors":"Nava Tintarev, Judith Masthoff","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wilson et al., 2003] Timothy D. Wilson, Jay Meyers, and Daniel T. Gilbert. \"How happy was I, anyway?\" A ret- rospective impact bias. Social Cognition, 21(6):421-446, 2003.","title":"How happy was I, anyway?\" A retrospective impact bias","authors":"Timothy D Wilson, Jay Meyers, Daniel T Gilbert","year":"2003","venue":"Social Cognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}