{"type":"ScholarlyArticle","title":"Beyond Strict Competition: Approximate Convergence of Multi-agent Q-Learning Dynamics","authors":[{"name":"Aamal Hussain","slug":"aamal-hussain","given":"Aamal","family":"Hussain","affiliation":"Imperial College London"},{"name":"Francesco Belardinelli","slug":"francesco-belardinelli","given":"Francesco","family":"Belardinelli","affiliation":"Imperial College London"},{"name":"Georgios Piliouras","slug":"georgios-piliouras","given":"Georgios","family":"Piliouras","affiliation":"Singapore University of Technology and Design"}],"abstract":"The behaviour of multi-agent learning in competitive settings is often considered under the restrictive assumption of a zero-sum game. Only under this strict requirement is the behaviour of learning well understood; beyond this, learning dynamics can often display non-convergent behaviours which prevent fixed-point analysis. Nonetheless, many relevant competitive games do not satisfy the zero-sum assumption.\r\nMotivated by this, we study a smooth variant of Q-Learning, a popular reinforcement learning dynamics which balances the agents' tendency to maximise their payoffs with their propensity to explore the state space. We examine this dynamic in games which are `close' to network zero-sum games and find that Q-Learning converges to a neighbourhood around a unique equilibrium. The size of the neighbourhood is determined by the `distance' to the zero-sum game, as well as the exploration rates of the agents. We complement these results by providing a method whereby, given an arbitrary network game, the `nearest' network zero-sum game can be found efficiently. Importantly, our theoretical guarantees are widely applicable in different game settings, regardless of whether the dynamics ultimately reach an equilibrium, or remain non convergent.","doi":"10.24963/ijcai.2023/16","url":"https://ijcai.org/proceedings/2023/16","pdf":"https://ijcai.org/proceedings/2023/0016.pdf","markdown":"https://ijcai.org/proceedings/2023/16.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/16","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":16,"pages":{"first":135,"last":143},"keywords":["Machine Learning: ML: Reinforcement learning","Game Theory and Economic Paradigms: GTEP: Noncooperative games","Agent-based and Multi-agent Systems: MAS: Multi-agent learning"],"videos":[],"references":[{"position":1,"raw":"[Abernethy et al., 2021] Jacob Abernethy, Kevin A Lai, and Andre Wibisono. Last-Iterate Convergence Rates for Min- Max Optimization: Convergence of Hamiltonian Gradient Descent and Consensus Optimization. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato, editors, Proceedings of the 32nd International Conference on Algorithmic Learn- ing Theory, volume 132 of Proceedings of Machine Learn- ing Research, pages 3-47. PMLR, 1 2021.","title":"Last-Iterate Convergence Rates for Min-Max Optimization: Convergence of Hamiltonian Gradient Descent and Consensus Optimization","authors":"Jacob Abernethy, Kevin A Lai, Andre Wibisono","year":"2021","venue":"Proceedings of the 32nd International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anagnostides et al., 2022] Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On Last-Iterate Convergence Beyond Zero-Sum Games. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learn- ing Research, pages 536-581. PMLR, 8 2022.","title":"On Last-Iterate Convergence Beyond Zero-Sum Games","authors":"Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, Tuomas Sandholm","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aumann, 1989] Robert J Aumann. Game Theory. In Game Theory, pages 1-53. Palgrave Macmillan UK, London, 1989.","title":"Game Theory","authors":"Robert J Aumann","year":"1989","venue":"Game Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aydın et al., 2022] Sarper Aydın, Sina Arefizadeh, and Cey- hun Eksin. Decentralized Fictitious Play in Near-Potential Games With Time-Varying Communication Networks. IEEE Control Systems Letters, 6:1226-1231, 2022.","title":"Decentralized Fictitious Play in Near-Potential Games With Time-Varying Communication Networks","authors":"Aydın","year":"2022","venue":"IEEE Control Systems Letters","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bailey and Piliouras, 2019a] James Bailey and Georgios Piliouras. Fast and furious learning in zero-sum games: Vanishing regret with non-vanishing step sizes. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural In- formation Processing Systems, volume 32.","title":"Fast and furious learning in zero-sum games: Vanishing regret with non-vanishing step sizes","authors":"Piliouras Bailey, James Bailey, Georgios Piliouras","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bailey and Piliouras, 2019b] James P Bailey and Georgios Piliouras. Multi-Agent Learning in Network Zero-Sum Games is a Hamiltonian System. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, pages 233-241, Rich- land, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems.","title":"Multi-Agent Learning in Network Zero-Sum Games is a Hamiltonian System","authors":"Piliouras ; Bailey, P James, Georgios Bailey, Piliouras","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cai et al., 2016] Yang Cai, Ozan Candogan, Constantinos Daskalakis, and Christos Papadimitriou. Zero-sum poly- matrix games: a generalization of minmax. Mathematics of Operations Research, 41(2):648-656, 5 2016.","title":"Zero-sum polymatrix games: a generalization of minmax","authors":"Yang Cai, Ozan Candogan, Constantinos Daskalakis, Christos Papadimitriou","year":"2016","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Candogan et al., 2013] Ozan Candogan, Asuman Ozdaglar, and Pablo A. Parrilo. Dynamics in near-potential games. Games and Economic Behavior, 82:66-90, 11 2013.","title":"Dynamics in near-potential games","authors":"Ozan Candogan, Asuman Ozdaglar, Pablo A Parrilo","year":"2013","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cheng and Ji, 2022] Daizhan Cheng and Zhengping Ji. Weighted and near weighted potential games with applica- tion to game theoretic control. Automatica, 141:110303, 7 2022.","title":"Weighted and near weighted potential games with application to game theoretic control","authors":"Ji ; Cheng, Daizhan Cheng, Zhengping Ji","year":"2022","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cheung and Tao, 2021] Yun Kuen Cheung and Yixin Tao. Chaos of learning beyond zero-sum and coordination via game decompositions. In International Conference on Learning Representations, 2021.","title":"Chaos of learning beyond zero-sum and coordination via game decompositions","authors":"Tao ; ] Cheung, Kuen Yun, Yixin Cheung, Tao","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ewerhart and Valkanova, 2020] Christian Ewerhart and Kremena Valkanova. Fictitious play in networks. Games and Economic Behavior, 123:182-206, 9 2020.","title":"Christian Ewerhart and Kremena Valkanova. Fictitious play in networks","authors":"Valkanova Ewerhart","year":"2020","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Galla and Farmer, 2013] Tobias Galla and J. Doyne Farmer. Complex dynamics in learning complicated games. Pro- ceedings of the National Academy of Sciences of the United States of America, 110(4):1232-1236, 2013.","title":"Complex dynamics in learning complicated games","authors":"Farmer ; Tobias Galla, J Doyne Galla, Farmer","year":"2013","venue":"Proceedings of the National Academy of Sciences of the United States of America","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Galla, 2011] Tobias Galla. Cycles of cooperation and defec- tion in imperfect learning. Journal of Statistical Mechan- ics: Theory and Experiment, 2011(8), 8 2011.","title":"Cycles of cooperation and defection in imperfect learning","authors":"Tobias Galla, Galla","year":"2011","venue":"Journal of Statistical Mechanics: Theory and Experiment","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gemp et al., 2022] Ian Gemp, Rahul Savani, Marc Lanc- tot, Yoram Bachrach, Thomas Anthony, Richard Ev- erett, Andrea Tacchetti, Tom Eccles, and János Kramár. Sample-based Approximation of Nash in Large Many- Player Games via Gradient Descent. In Proceedings of the 21st International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '22, pages 507-515, Rich- land, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems.","title":"Sample-based Approximation of Nash in Large Many-Player Games via Gradient Descent","authors":"Ian Gemp, Rahul Savani, Marc Lanctot, Yoram Bachrach, Thomas Anthony, Richard Everett, Andrea Tacchetti, Tom Eccles, János Kramár","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '22","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hadikhanloo et al., 2022] Saeed Hadikhanloo, Rida Laraki, Panayotis Mertikopoulos, and Sylvain Sorin. Learning in nonatomic games part I Finite action spaces and pop- ulation games. Journal of Dynamics and Games. 2022, 0(0):0, 2022.","title":"Learning in nonatomic games part I Finite action spaces and population games","authors":"Saeed Hadikhanloo, Rida Laraki, Panayotis Mertikopoulos, Sylvain Sorin","year":"2022","venue":"Journal of Dynamics and Games","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Harris, 1998] Christopher Harris. On the Rate of Conver- gence of Continuous-Time Fictitious Play. Games and Economic Behavior, 22(2):238-259, 2 1998.","title":"On the Rate of Convergence of Continuous-Time Fictitious Play","authors":"Christopher Harris","year":"1998","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hofbauer and Sorin, 2005] Josef Hofbauer and Sylvain Sorin. Best response dynamics for continuous zero-sum games. Discrete and Continuous Dynamical Systems -B. 2006, Volume 6, Pages 215-224, 6(1):215, 10 2005.","title":"Best response dynamics for continuous zero-sum games","authors":"Sorin Hofbauer, Josef Hofbauer, Sylvain Sorin","year":"2005","venue":"Discrete and Continuous Dynamical Systems -B","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hofbauer, 1996] Josef Hofbauer. Evolutionary dynamics for bimatrix games: A Hamiltonian system? Journal of Mathematical Biology, 34(5-6):675-688, 1996.","title":"Evolutionary dynamics for bimatrix games: A Hamiltonian system","authors":"Josef Hofbauer, Hofbauer","year":"1996","venue":"Journal of Mathematical Biology","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kianercy and Galstyan, 2012] Ardeshir Kianercy and Aram Galstyan. Dynamics of Boltzmann Q learning in two- player two-action games. Physical Review E -Statistical, Nonlinear, and Soft Matter Physics, 85(4):041145, 4 2012.","title":"Ardeshir Kianercy and Aram Galstyan. Dynamics of Boltzmann Q learning in twoplayer two-action games","authors":"Galstyan Kianercy","year":"2012","venue":"Physical Review E -Statistical, Nonlinear, and Soft Matter Physics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Leonardos and Piliouras, 2022] Stefanos Leonardos and Georgios Piliouras. Exploration-exploitation in multi- agent learning: Catastrophe theory meets game theory. Artificial Intelligence, 304:103653, 2022.","title":"Exploration-exploitation in multiagent learning: Catastrophe theory meets game theory","authors":"Leonardos, Piliouras, Stefanos Leonardos, Georgios Piliouras","year":"2022","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Leonardos et al., 2021] Stefanos Leonardos, Georgios Pil- iouras, and Kelly Spendlove. Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationality. Advances in Neural Information Processing Systems, 34:26318-26331, 12 2021.","title":"Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationality","authors":"Stefanos Leonardos, Georgios Piliouras, Kelly Spendlove","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2018] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Land- scape of Neural Nets. In Advances in Neural Information Processing Systems, 2018.","title":"Visualizing the Loss Landscape of Neural Nets","authors":"Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[McKelvey and Palfrey, 1995] Richard D. McKelvey and Thomas R. Palfrey. Quantal Response Equilibria for Normal Form Games. Games and Economic Behavior, 10(1):6-38, 7 1995.","title":"Quantal Response Equilibria for Normal Form Games","authors":"Richard D Mckelvey, Thomas R Palfrey","year":"1995","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Melo, 2021] Emerson Melo. On the Uniqueness of Quan- tal Response Equilibria and Its Application to Network Games. SSRN Electronic Journal, 6 2021.","title":"On the Uniqueness of Quantal Response Equilibria and Its Application to Network Games","authors":"Emerson Melo","year":"2021","venue":"SSRN Electronic Journal","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Mertikopoulos et al., ] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in Adver- sarial Regularized Learning.","title":"Cycles in Adversarial Regularized Learning","authors":"Mertikopoulos","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Metrick and Polak, 1994] Andrew I Metrick and Ben Polak. Fictitious play in 2 • 2 games: a geometric proof of con- vergence*. Econ. Theory, 4:923-933, 1994.","title":"Fictitious play in 2 • 2 games: a geometric proof of convergence","authors":"Polak ; Metrick, I Andrew, Ben Metrick, Polak","year":"1994","venue":"Econ. Theory","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Monderer and Shapley, 1996] Dov Monderer and Lloyd S. Shapley. Potential games. Games and Economic Behavior, 14(1):124-143, 5 1996.","title":"Potential games","authors":"Shapley Monderer, Dov Monderer, Lloyd S Shapley","year":"1996","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mukhopadhyay and Chakraborty, 2020] Archan Mukhopadhyay and Sagar Chakraborty. Deciphering chaos in evolutionary games. Chaos, 30(12):121104, 12 2020.","title":"Deciphering chaos in evolutionary games","authors":"Archan Mukhopadhyay, Sagar Chakraborty","year":null,"venue":"Chaos","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Nisan et al., 2007] Noam Nisan, Tim Roughgarden, Éva Tardos, and Vijay V. Vazirani. Algorithmic Game Theory. Cambridge University Press, New York, NY, USA, 2007.","title":"Algorithmic Game Theory","authors":"Nisan","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Roberson, 2006] Brian Roberson. The Colonel Blotto game. Economic Theory, 29(1):1-24, 2006.","title":"The Colonel Blotto game","authors":"Brian Roberson","year":"2006","venue":"Economic Theory","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sanders et al., 2018] James B T Sanders, J Doyne Farmer, and Tobias Galla. The prevalence of chaotic dynamics in games with many players. Scientific Reports, 8(1):4902, 2018.","title":"The prevalence of chaotic dynamics in games with many players","authors":"B T James, J Doyne Sanders, Tobias Farmer, Galla","year":"2018","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sato and Crutchfield, 2003] Yuzuru Sato and James P. Crutchfield. Coupled replicator equations for the dynam- ics of learning in multiagent systems. Physical Review E, 67(1):015206, 1 2003.","title":"Coupled replicator equations for the dynamics of learning in multiagent systems","authors":"Yuzuru Sato, James P Crutchfield","year":null,"venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sato et al., 2002] Yuzuru Sato, Eizo Akiyama, and J. Doyne Farmer. Chaos in learning a simple two-person game. Proceedings of the National Academy of Sciences of the United States of America, 99(7):4748-4751, 4 2002.","title":"Chaos in learning a simple two-person game","authors":"Yuzuru Sato, Eizo Akiyama, J Doyne Farmer","year":null,"venue":"Proceedings of the National Academy of Sciences of the United States of America","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Schwartz, 2014] Howard M. Schwartz. Multi-Agent Ma- chine Learning: A Reinforcement Approach. Wiley, 2014.","title":"Multi-Agent Machine Learning: A Reinforcement Approach","authors":"M Howard, Schwartz","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sutton and Barto, 2018] R Sutton and A Barto. Reinforce- ment Learning: An Introduction. MIT Press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"R Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tuyls et al., 2006] Karl Tuyls, Pieter Jan 'T Hoen, and Bram Vanschoenwinkel. An evolutionary dynamical anal- ysis of multi-agent learning in iterated games. Autonomous Agents and Multi-Agent Systems, 12:115-153, 2006.","title":"An evolutionary dynamical analysis of multi-agent learning in iterated games","authors":"Tuyls","year":"2006","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Vlatakis-Gkaragkounis et al., 2020] Emmanouil-Vasileios Vlatakis-Gkaragkounis, Lampros Flokas, Thanasis Lianeas, Panayotis Mertikopoulos, and Georgios Pil- iouras. No-Regret Learning and Mixed Nash Equilibria: They Do Not Mix. Advances in Neural Information Processing Systems, 33:1380-1391, 2020.","title":"No-Regret Learning and Mixed Nash Equilibria: They Do Not Mix","authors":"Emmanouil-Vasileios Vlatakis-Gkaragkounis, Lampros Flokas, Thanasis Lianeas, Panayotis Mertikopoulos, Georgios Piliouras","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}