{"type":"ScholarlyArticle","title":"Black-box Audit of YouTube's Video Recommendation: Investigation of Misinformation Filter Bubble Dynamics (Extended Abstract)","authors":[{"name":"Matus Tomlein","slug":"matus-tomlein","given":"Matus","family":"Tomlein","affiliation":"Kempelen Institute of Intelligent Technologies"},{"name":"Branislav Pecher","slug":"branislav-pecher","given":"Branislav","family":"Pecher","affiliation":"Kempelen Institute of Intelligent Technologies; Faculty of Information Technology , Brno University of Technology"},{"name":"Jakub Simko","slug":"jakub-simko","given":"Jakub","family":"Simko","affiliation":"Kempelen Institute of Intelligent Technologies"},{"name":"Ivan Srba","slug":"ivan-srba","given":"Ivan","family":"Srba","affiliation":"Kempelen Institute of Intelligent Technologies"},{"name":"Robert Moro","slug":"robert-moro","given":"Robert","family":"Moro","affiliation":"Kempelen Institute of Intelligent Technologies"},{"name":"Elena Stefancova","slug":"elena-stefancova","given":"Elena","family":"Stefancova","affiliation":"Kempelen Institute of Intelligent Technologies"},{"name":"Michal Kompan","slug":"michal-kompan","given":"Michal","family":"Kompan","affiliation":"Kempelen Institute of Intelligent Technologies; Slovak Centre for Research of Artificial Intelligence -slovak.AI"},{"name":"Andrea Hrckova","slug":"andrea-hrckova","given":"Andrea","family":"Hrckova","affiliation":"Kempelen Institute of Intelligent Technologies"},{"name":"Juraj Podrouzek","slug":"juraj-podrouzek","given":"Juraj","family":"Podrouzek","affiliation":"Kempelen Institute of Intelligent Technologies"},{"name":"Maria Bielikova","slug":"maria-bielikova","given":"Maria","family":"Bielikova","affiliation":"Kempelen Institute of Intelligent Technologies; Slovak Centre for Research of Artificial Intelligence -slovak.AI"}],"abstract":"In this paper, we describe a black-box sockpuppeting audit which we carried out to investigate the creation and bursting dynamics of misinformation filter bubbles on YouTube. Pre-programmed agents acting as YouTube users stimulated YouTube's recommender systems: they first watched a series of misinformation promoting videos (bubble creation) and then a series of misinformation debunking videos (bubble bursting). Meanwhile, agents logged videos recommended to them by YouTube. After manually annotating these recommendations, we were able to quantify the portion of misinformative videos among them. The results confirm the creation of filter bubbles (albeit not in all situations) and show that these bubbles can be bursted by watching credible content. Drawing a direct comparison with a previous study, we do not see improvements in overall quantities of misinformation recommended.","doi":"10.24963/ijcai.2022/749","url":"https://ijcai.org/proceedings/2022/749","pdf":"https://ijcai.org/proceedings/2022/0749.pdf","markdown":"https://ijcai.org/proceedings/2022/749.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/749","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":749,"pages":{"first":5349,"last":5353},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987261","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985404","length_sec":673,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abul-Fottouh et al., 2020] Deena Abul-Fottouh, Melodie Yunju Song, and Anatoliy Gruzd. Examin- ing algorithmic biases in youtube's recommendations of vaccine videos. Int. Journal of Medical Informatics, 140:104175, 2020.","title":"Examining algorithmic biases in youtube's recommendations of vaccine videos","authors":"Deena Abul-Fottouh, Melodie Yunju Song, Anatoliy Gruzd","year":"2020","venue":"Int. Journal of Medical Informatics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[AI HLEG, 2019] AI HLEG. Ethics Guidelines for Trust- worthy AI. Technical report, European Commission, 2019.","title":"AI HLEG. Ethics Guidelines for Trustworthy AI","authors":"A I Hleg","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hannak et al., 2013] Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. Measuring per- sonalization of web search. In Proc. of the 22nd Interna- tional Conference on World Wide Web (WWW '13), page 527-538, New York, NY, USA, 2013. ACM.","title":"Measuring personalization of web search","authors":"Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, Christo Wilson","year":"2013","venue":"Proc. of the 22nd International Conference on World Wide Web (WWW '13)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hou et al., 2019] Rui Hou, Veronica Perez-Rosas, Stacy Loeb, and Rada Mihalcea. Towards automatic detection of misinformation in online medical videos. In 2019 In- ternational Conference on Multimodal Interaction, ICMI '19, page 235-243, New York, NY, USA, 2019. Associa- tion for Computing Machinery.","title":"Towards automatic detection of misinformation in online medical videos","authors":"Rui Hou, Veronica Perez-Rosas, Stacy Loeb, Rada Mihalcea","year":"2019","venue":"2019 International Conference on Multimodal Interaction, ICMI '19","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hussein et al., 2020] Eslam Hussein, Prerna Juneja, and Tanushree Mitra. Measuring misinformation in video search platforms: An audit study on youtube. Proc. ACM Hum.-Comput. Interact., 4(CSCW1), May 2020.","title":"Measuring misinformation in video search platforms: An audit study on youtube","authors":"Eslam Hussein, Prerna Juneja, Tanushree Mitra","year":"2020","venue":"Proc. ACM Hum.-Comput. Interact","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Juneja and Mitra, 2021] Prerna Juneja and Tanushree Mi- tra. Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation. In Proc. of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21), 2021.","title":"Prerna Juneja and Tanushree Mitra. Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation","authors":"Mitra Juneja","year":"2021","venue":"Proc. of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Le et al., 2019] Huyen Le, Andrew High, Raven Maragh, Timothy Havens, Brian Ekdale, and Zubair Shafiq. Mea- suring political personalization of Google news search. In Proc. of the World Wide Web Conference (WWW '19), pages 2957-2963, 2019.","title":"Measuring political personalization of Google news search","authors":"Huyen Le, Andrew High, Raven Maragh, Timothy Havens, Brian Ekdale, Zubair Shafiq","year":"2019","venue":"Proc. of the World Wide Web Conference (WWW '19)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Metaxa et al., 2019] Danaë Metaxa, Joon Sung Park, James A. Landay, and Jeff Hancock. Search media and elections: A longitudinal investigation of political search results. Proc. ACM Hum.-Comput. Interact., 3(CSCW), November 2019.","title":"Search media and elections: A longitudinal investigation of political search results","authors":"Danaë Metaxa, Sung Joon, James A Park, Jeff Landay, Hancock","year":"2019","venue":"Proc. ACM Hum.-Comput. Interact","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Papadamou et al., 2020] Kostantinos Papadamou, Savvas Zannettou, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Michael Sirivianos. \"it is just a flu\": Assessing the effect of watch history on youtube's pseudoscientific video recommendations. In Proceedings of the Sixteenth International AAAI Conference on Web and Social Media (ICWSM-2022), Palo Alto, California, USA, 2020. AAAI Press. [to appear].","title":"it is just a flu\": Assessing the effect of watch history on youtube's pseudoscientific video recommendations","authors":"Kostantinos Papadamou, Savvas Zannettou, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Michael Sirivianos","year":"2020","venue":"Proceedings of the Sixteenth International AAAI Conference on Web and Social Media (ICWSM-2022)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ribeiro et al., 2020] Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgílio A. F. Almeida, and Wagner Meira. Auditing radicalization pathways on youtube. In Proc. of the 2020 Conf. on Fairness, Accountability, and Trans- parency, page 131-141, New York, NY, USA, 2020. ACM.","title":"Auditing radicalization pathways on youtube","authors":"Ribeiro","year":"2020","venue":"Proc. of the 2020 Conf. on Fairness, Accountability, and Transparency","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Robertson et al., 2018] Ronald E. Robertson, David Lazer, and Christo Wilson. Auditing the personalization and composition of politically-related search engine results pages. In Proc. of the 2018 World Wide Web Conf., WWW '18, page 955-965, Republic and Canton of Geneva, CHE, 2018. Int. WWW Conferences Steering Committee.","title":"Auditing the personalization and composition of politically-related search engine results pages","authors":"Ronald E Robertson, David Lazer, Christo Wilson","year":"2018","venue":"Proc. of the 2018 World Wide Web Conf., WWW '18","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sandvig et al., 2014] Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. Auditing algo- rithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry, 22:4349-4357, 2014.","title":"Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive","authors":"Sandvig","year":"2014","venue":"inquiry","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Silva et al., 2020] Márcio Silva, Lucas Santos de Oliveira, Athanasios Andreou, Pedro Olmo Vaz de Melo, Oana Goga, and Fabricio Benevenuto. Facebook ads monitor: An independent auditing system for political ads on face- book. In Proc. of The Web Conference (WWW '20), page 224-234, New York, NY, USA, 2020. ACM.","title":"Facebook ads monitor: An independent auditing system for political ads on facebook","authors":"Márcio Silva, Lucas Santos De Oliveira, Athanasios Andreou, Pedro Olmo Vaz De Melo, Oana Goga, Fabricio Benevenuto","year":"2020","venue":"Proc. of The Web Conference (WWW '20)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Simko et al., 2021] Jakub Simko, Matus Tomlein, Branislav Pecher, Robert Moro, Ivan Srba, Elena Stefancova, Andrea Hrckova, Michal Kompan, Juraj Podrouzek, and Maria Bielikova. Towards continuous automatic audits of so- cial media adaptive behavior and its role in misinforma- tion spreading. In Adjunct Proc. of the 29th ACM Conf. on User Modeling, Adaptation and Personalization (UMAP '21), page 411-414, New York, NY, USA, 2021. ACM.","title":"Towards continuous automatic audits of social media adaptive behavior and its role in misinformation spreading","authors":"Jakub Simko, Matus Tomlein, Branislav Pecher, Robert Moro, Ivan Srba, Elena Stefancova, Andrea Hrckova, Michal Kompan, Juraj Podrouzek, Maria Bielikova","year":"2021","venue":"Adjunct Proc. of the 29th ACM Conf. on User Modeling, Adaptation and Personalization (UMAP '21)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Spinelli and Crovella, 2020] Larissa Spinelli and Mark Crovella. How youtube leads privacy-seeking users away from reliable information. In Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization, page 244-251, New York, NY, USA, 2020. ACM.","title":"How youtube leads privacy-seeking users away from reliable information","authors":"Crovella Spinelli, Larissa Spinelli, Mark Crovella","year":"2020","venue":"Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tomlein et al., 2021] Matus Tomlein, Branislav Pecher, Jakub Simko, Ivan Srba, Robert Moro, Elena Stefancova, Michal Kompan, Andrea Hrckova, Juraj Podrouzek, and Maria Bielikova. An Audit of Misinformation Filter Bub- bles on YouTube: Bubble Bursting and Recent Behavior Changes. In Fifteenth ACM Conference on Recommender Systems, pages 1-11, New York, NY, USA, September 2021. ACM.","title":"An Audit of Misinformation Filter Bubbles on YouTube: Bubble Bursting and Recent Behavior Changes","authors":"Matus Tomlein, Branislav Pecher, Jakub Simko, Ivan Srba, Robert Moro, Elena Stefancova, Michal Kompan, Andrea Hrckova, Juraj Podrouzek, Maria Bielikova","year":"2021","venue":"Fifteenth ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vaidhyanathan, 2018] Siva Vaidhyanathan. Antisocial me- dia: How Facebook disconnects us and undermines democracy. Oxford University Press, 2018.","title":"Antisocial media: How Facebook disconnects us and undermines democracy","authors":"Siva Vaidhyanathan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[YouTube, 2020] YouTube. Managing harmful con- spiracy theories on youtube. https://blog.youtube/ news-and-events/harmful-conspiracy-theories-youtube/, 2020. Accessed: 2022-06-03.","title":"YouTube. Managing harmful conspiracy theories on youtube","authors":null,"year":"2020","venue":null,"doi":null,"url":"https://blog.youtube/news-and-events/harmful-conspiracy-theories-youtube/,2020","internal_url":null},{"position":19,"raw":"[Zuboff, 2019] Shoshana Zuboff. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. Profile Books, 2019.","title":"The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power","authors":"Shoshana Zuboff","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}