{"type":"ScholarlyArticle","title":"By Fair Means or Foul: Quantifying Collusion in a Market Simulation with Deep Reinforcement Learning","authors":[{"name":"Michael Schlechtinger","slug":"michael-schlechtinger","given":"Michael","family":"Schlechtinger","affiliation":"University of Mannheim - Chair of Data Science"},{"name":"Damaris Kosack","slug":"damaris-kosack","given":"Damaris","family":"Kosack","affiliation":"University of Mannheim -Chair of Law {schlechtinger ,"},{"name":"Franz Krause","slug":"franz-krause","given":"Franz","family":"Krause","affiliation":"University of Mannheim - Chair of Data Science"},{"name":"Heiko Paulheim","slug":"heiko-paulheim","given":"Heiko","family":"Paulheim","affiliation":"University of Mannheim - Chair of Data Science"}],"abstract":"In the rapidly evolving landscape of eCommerce, Artificial Intelligence (AI) based pricing algorithms, particularly those utilizing Reinforcement Learning (RL), are becoming increasingly prevalent. This rise has led to an inextricable pricing situation with the potential for market collusion. Our research employs an experimental oligopoly model of repeated price competition, systematically varying the environment to cover scenarios from basic economic theory to subjective consumer demand preferences. We also introduce a novel demand framework that enables the implementation of various demand models, allowing for a weighted blending of different models.\r\nIn contrast to existing research in this domain, we aim to investigate the strategies and emerging pricing patterns developed by the agents, which may lead to a collusive outcome. Furthermore, we investigate a scenario where agents cannot observe their competitors’ prices. Finally, we provide a comprehensive legal analysis across all scenarios. \r\nOur findings indicate that RL-based AI agents converge to a collusive state characterized by the charging of supracompetitive prices, without necessarily requiring inter-agent communication. Implementing alternative RL algorithms, altering the number of agents or simulation settings, and restricting the scope of the agents’ observation space does not significantly impact the collusive market outcome behavior.","doi":"10.24963/ijcai.2024/54","url":"https://ijcai.org/proceedings/2024/54","pdf":"https://ijcai.org/proceedings/2024/0054.pdf","markdown":"https://ijcai.org/proceedings/2024/54.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/54","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI Ethics, Trust, Fairness","track":"Main Track","number":54,"pages":{"first":485,"last":493},"keywords":["AI Ethics, Trust, Fairness: ETF: AI and law, governance, regulation","Machine Learning: ML: Reinforcement learning","Multidisciplinary Topics and Applications: MTA: Economics","Agent-based and Multi-agent Systems: MAS: Agent-based simulation and emergence"],"videos":[],"references":[{"position":1,"raw":"The Notifications, Time Limits and Hearings Provided for in Council Regulation (EEC), March 1998.","title":"The Notifications","authors":null,"year":"1998","venue":"Time Limits and Hearings Provided for in Council Regulation (EEC)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abada and Lambin, 2023] Ibrahim Abada and Xavier Lam- bin. Artificial Intelligence: Can Seemingly Collusive Out- comes Be Avoided? Management Science, 69(9):5042- 5065, September 2023.","title":"Artificial Intelligence: Can Seemingly Collusive Outcomes Be Avoided?","authors":"Lambin Abada","year":"2023","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Achiam, 2018] Joshua Achiam. Spinning Up in Deep Rein- forcement Learning. https://spinningup.openai.com/, Jan- uary 2018.","title":"Joshua Achiam. Spinning Up in Deep Reinforcement Learning","authors":"Achiam","year":"2018","venue":null,"doi":null,"url":"https://spinningup.openai.com/","internal_url":null},{"position":4,"raw":"[Assad et al., 2020] Stephanie Assad, Robert Clark, Daniel Ershov, and Lei Xu. Algorithmic Pricing and Competi- tion: Empirical Evidence from the German Retail Gaso- line Market. CESifo Working Paper Series, 1(8521), Jan- uary 2020.","title":"Algorithmic Pricing and Competition: Empirical Evidence from the German Retail Gasoline Market","authors":"Stephanie Assad, Robert Clark, Daniel Ershov, Lei Xu","year":"2020","venue":"CESifo Working Paper Series","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bertrand, 1883] Joseph Bertrand. Théorie des Richesses: Revue de Théories mathématiques de la richesse so- ciale par Léon Walras et Recherches sur les principes mathématiques de la théorie des richesses par Augustin Cournot. Journal des Savants, page 499, January 1883.","title":"Théorie des Richesses: Revue de Théories mathématiques de la richesse sociale par Léon Walras et Recherches sur les principes mathématiques de la théorie des richesses par Augustin Cournot","authors":"Joseph Bertrand","year":"1883","venue":"Journal des Savants","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown and MacKay, 2021] Zach Brown and Alexander MacKay. Competition in Pricing Algorithms. National Bureau of Economic Research, January 2021.","title":"Competition in Pricing Algorithms","authors":"Mackay Brown, Zach Brown, Alexander Mackay","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Calvano et al., 2018] Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolo, and Sergio Pastorello. Artificial In- telligence, Algorithmic Pricing and Collusion. American Economic Review, 110(10):3267-97, January 2018.","title":"Artificial Intelligence, Algorithmic Pricing and Collusion","authors":"Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolo, Sergio Pastorello","year":"2018","venue":"American Economic Review","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[ECJ, 1998] ECJ. Case C-7-/95 -Deere/Kommission, Jan- uary 1998.","title":"ECJ. Case C-7-/95 -Deere/Kommission","authors":"Ecj","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[ECJ, 1999] ECJ. Case C-199/92 -Hüls v Commission, Jan- uary 1999.","title":"ECJ. Case C-199/92 -Hüls v Commission","authors":"Ecj","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[ECJ, 2009] ECJ. Case C-8/08 -T-Mobile Netherlands v Raad van Bestuur van de Nederlandse Mededing- ingsautoriteit, January 2009.","title":"ECJ. Case C-8/08 -T-Mobile Netherlands v Raad van Bestuur van de Nederlandse Mededingingsautoriteit","authors":"Ecj","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[ECJ, 2015] ECJ. Case C-286/13 -Dole Food and Dole Fresh Fruit Europe, January 2015. [ECJ, 2016] ECJ. Case -74/14 -Eturas UAB v Lietuvos Re- spublikos konkurencijos taryba, January 2016.","title":"ECJ. Case C-286/13 -Dole Food and Dole Fresh Fruit Europe","authors":"Ecj","year":"2015","venue":"ECJ. Case -74/14","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han, 2021] Bingyan Han. Understanding al- gorithmic collusion with experience replay. https://arxiv.org/pdf/2102.09139, February 2021.","title":"Understanding algorithmic collusion with experience replay","authors":"Han ;, Bingyan Han","year":"2021","venue":null,"doi":null,"url":"https://arxiv.org/pdf/2102.09139","internal_url":null},{"position":13,"raw":"[Hansen et al., 2020] Karsten Hansen, Kanishka Misra, and Mallesh Pai. Algorithmic Collusion: Supra-competitive Prices via Independent Algorithms. CEPR Discussion Pa- pers, 40(1)(14372), January 2020.","title":"Algorithmic Collusion: Supra-competitive Prices via Independent Algorithms","authors":"Karsten Hansen, Kanishka Misra, Mallesh Pai","year":"2020","venue":"CEPR Discussion Papers","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hausknecht et al., 2015] Matthew Hausknecht, Pe- ter Stone, and arXiv:1507.06527. Deep Recur- rent Q-Learning for Partially Observable MDPs. https://arxiv.org/pdf/1507.06527, July 2015.","title":"Deep Recurrent Q-Learning for Partially Observable MDPs","authors":"Matthew Hausknecht, Peter Stone","year":"2015","venue":null,"doi":null,"url":"https://arxiv.org/pdf/1507.06527","internal_url":null},{"position":15,"raw":"[Hettich, 2021] Matthias Hettich. Algorithmic Collusion: Insights from Deep Learning. SSRN Electronic Journal, January 2021.","title":"Algorithmic Collusion: Insights from Deep Learning","authors":"Matthias Hettich","year":"2021","venue":"SSRN Electronic Journal","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kimbrough and Murphy, 2009] Steven O. Kimbrough and Frederic H. Murphy. Learning to Collude Tacitly on Pro- duction Levels by Oligopolistic Agents. Computational Economics, 33(1):47-78, January 2009.","title":"Learning to Collude Tacitly on Production Levels by Oligopolistic Agents","authors":"Steven O Kimbrough, Frederic H Murphy","year":"2009","venue":"Computational Economics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Klein, 2021] Timo Klein. Autonomous algorithmic collu- sion: Q-learning under sequential pricing. The RAND Journal of Economics, 52(3):538-558, January 2021.","title":"Autonomous algorithmic collusion: Q-learning under sequential pricing","authors":"Timo Klein","year":"2021","venue":"The RAND Journal of Economics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kropp et al., 2019] Lennard Alexander Kropp, Jakob J. Ko- rbel, M. Theilig, and R. Zarnekow. Dynamic Pricing of Product Clusters: A Multi-Agent Reinforcement Learning Approach. In Proceedings of the 27th European Confer- ence on Information Systems (ECIS), volume 27, Stock- holm & Uppsala, Sweden, January 2019. Association for Information Systems.","title":"Dynamic Pricing of Product Clusters: A Multi-Agent Reinforcement Learning Approach","authors":"Lennard Alexander Kropp, Jakob J Korbel, M Theilig, R Zarnekow","year":"2019","venue":"Proceedings of the 27th European Conference on Information Systems (ECIS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, and arXiv:1312.5602. Play- ing Atari with Deep Reinforcement Learning. In Proceed- ings of the 26th International Conference on Neural Infor- mation Processing Systems, volume 2.","title":"Playing Atari with Deep Reinforcement Learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":null,"venue":"Proceedings of the 26th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Musolff, 2022] Leon Musolff. Algorithmic Pricing Facil- itates Tacit Collusion: Evidence from E-Commerce. In Proceedings of the 23rd ACM Conference on Economics and Computation, EC '22, pages 32-33, New York, NY, USA, July 2022. Association for Computing Machinery.","title":"Algorithmic Pricing Facilitates Tacit Collusion: Evidence from E-Commerce","authors":"Leon Musolff","year":"2022","venue":"Proceedings of the 23rd ACM Conference on Economics and Computation, EC '22","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qiao et al., 2024] Wenchuan Qiao, Min Huang, Zheming Gao, and Xingwei Wang. Distributed dynamic pricing of multiple perishable products using multi-agent rein- forcement learning. Expert Systems with Applications, 237:121252, March 2024.","title":"Distributed dynamic pricing of multiple perishable products using multi-agent reinforcement learning","authors":"Qiao","year":"2024","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Raffin et al., 2021] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268):1-8, January 2021.","title":"Stable-Baselines3: Reliable Reinforcement Learning Implementations","authors":"Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, Noah Dormann","year":"2021","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Royston, 1992] Patrick Royston. Lowess Smoothing. Stata Technical Bulletin, 1(3), January 1992.","title":"Lowess Smoothing","authors":"Patrick Royston","year":"1992","venue":"Stata Technical Bulletin","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schlechtinger et al., 2021] Michael Schlechtinger, Damaris Kosack, Heiko Paulheim, and Thomas Fetzer. Winning at Any Cost -Infringing the Cartel Prohibition With Re- inforcement Learning. In Advances in Practical Applica- tions of Agents, Multi-Agent Systems, and Social Good. The PAAMS Collection, Salamanca, Spain, January 2021. Springer Cham. [Schulman et al., 2017]","title":"Winning at Any Cost -Infringing the Cartel Prohibition With Reinforcement Learning","authors":"Michael Schlechtinger, Damaris Kosack, Heiko Paulheim, Thomas Fetzer","year":"2017","venue":"Advances in Practical Applications of Agents, Multi-Agent Systems, and Social Good","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, Oleg Klimov, and arXiv:1707.06347. Proximal Policy Optimization Algo- rithms. https://arxiv.org/pdf/1707.06347, July 2017.","title":"Proximal Policy Optimization Algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":"https://arxiv.org/pdf/1707.06347","internal_url":null},{"position":26,"raw":"[Siallagan et al., 2013] Manahan Siallagan, Hiroshi Deguchi, and Manabu Ichikawa. Aspiration-Based Learning in a Cournot Duopoly Model. Evolutionary and Institutional Economics Review, 10(2):295-314, January 2013.","title":"Aspiration-Based Learning in a Cournot Duopoly Model","authors":"Manahan Siallagan, Hiroshi Deguchi, Manabu Ichikawa","year":"2013","venue":"Evolutionary and Institutional Economics Review","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[van Otterlo and Wiering, 2012] Martijn van Otterlo and Marco Wiering. Reinforcement Learning and Markov De- cision Processes. In Marco Wiering and Martijn van Ot- terlo, editors, Reinforcement Learning: State-of-the-art, volume v. 12 of Adaptation, Learning, and Optimization, pages 3-42. Springer, Heidelberg; New York, January 2012.","title":"Reinforcement Learning and Markov Decision Processes","authors":"Van Otterlo, Marco Wiering ; Martijn Van Otterlo, Wiering","year":"2012","venue":"Reinforcement Learning: State-of-the-art, volume v. 12 of Adaptation, Learning, and Optimization","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Waltman and Kaymak, 2008] Ludo Waltman and Uzay Kaymak. Q-learning agents in a Cournot oligopoly model. Journal of Economic Dynamics and Control, 32(10):3275-3293, January 2008.","title":"Q-learning agents in a Cournot oligopoly model","authors":"Kaymak Waltman, Ludo Waltman, Uzay Kaymak","year":"2008","venue":"Journal of Economic Dynamics and Control","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Werner, 2021] Tobias Werner. Algorithmic and Human Col- lusion. Heinrich Heine University Düsseldorf, January 2021.","title":"Algorithmic and Human Collusion","authors":"Werner","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhong et al., 2005] Jinghui Zhong, Xiaomin Hu, Jun Zhang, and Min Gu. Comparison of Performance between Different Selection Strategies on Simple Genetic Algo- rithms. In International Conference on Computational In- telligence for Modelling, Control and Automation and In- ternational Conference on Intelligent Agents, Web Tech- nologies and Internet Commerce (CIMCA/IAWTIC'06), pages 1115-1121, Vienna, Austria, 2005. IEEE.","title":"Comparison of Performance between Different Selection Strategies on Simple Genetic Algorithms","authors":"Jinghui Zhong, Xiaomin Hu, Jun Zhang, Min Gu","year":"2005","venue":"International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA/IAWTIC'06)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}