{"type":"ScholarlyArticle","title":"Modelling the Dynamics of Multi-Agent Q-learning: The Stochastic Effects of Local Interaction and Incomplete Information","authors":[{"name":"Chin-wing Leung","slug":"chin-wing-leung","given":"Chin-wing","family":"Leung","affiliation":"The Chinese University of Hong Kong"},{"name":"Shuyue Hu","slug":"shuyue-hu","given":"Shuyue","family":"Hu","affiliation":"Shanghai Artificial Intelligence Laboratory"},{"name":"Ho-fung Leung","slug":"ho-fung-leung","given":"Ho-fung","family":"Leung","affiliation":"The Chinese University of Hong Kong"}],"abstract":"The theoretical underpinnings of multiagent reinforcement learning has recently attracted much attention.\r\nIn this work, we focus on the generalized social learning (GSL) protocol --- an agent interaction protocol that is widely adopted in the literature, and aim to develop an accurate theoretical model for the Q-learning dynamics under this protocol.\r\nNoting that previous models fail to characterize the effects of local interactions and incomplete information that arise from GSL, we model the Q-values dynamics of each individual agent as a system of stochastic differential equations (SDE). \r\nBased on the SDE, we express the time evolution of the probability density function of Q-values in the population with a Fokker-Planck equation.\r\nWe validate the correctness of our model through extensive comparisons with agent-based simulation results across different types of symmetric games. \r\nIn addition, we show that as the interactions between agents are more limited and information is less complete, the population can converge to a outcome that is qualitatively different than that with global interactions and complete information.","doi":"10.24963/ijcai.2022/55","url":"https://ijcai.org/proceedings/2022/55","pdf":"https://ijcai.org/proceedings/2022/0055.pdf","markdown":"https://ijcai.org/proceedings/2022/55.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/55","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":55,"pages":{"first":384,"last":390},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence","Agent-based and Multi-agent Systems: Agent Theories and Models"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987110","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985253","length_sec":714,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anastassacos et al., 2020] Nicolas Anastassacos, Stephen Hailes, and Mirco Musolesi. Partner selection for the emergence of cooperation in multi-agent systems using re- inforcement learning. In AAAI, pages 7047-7054, 2020.","title":"Partner selection for the emergence of cooperation in multi-agent systems using reinforcement learning","authors":"Anastassacos","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker, 2020] Bowen Baker. Emergent reciprocity and team formation from randomized uncertain social preferences. arXiv preprint arXiv:2011.05373, 2020.","title":"Emergent reciprocity and team formation from randomized uncertain social preferences","authors":"Bowen Baker, Baker","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bloembergen et al., 2015] Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dy- namics of multi-agent learning: A survey. Journal of Arti- ficial Intelligence Research, 53:659-697, 2015.","title":"Evolutionary dynamics of multi-agent learning: A survey","authors":"Daan Bloembergen, Karl Tuyls, Daniel Hennes, Michael Kaisers","year":"2015","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Campbell and Wu, 2011] Adam Campbell and Annie S Wu. Multi-agent role allocation: issues, approaches, and multi- ple perspectives. AAMAS, 22(2):317-355, 2011.","title":"Multi-agent role allocation: issues, approaches, and multiple perspectives","authors":"Wu Campbell, Adam Campbell, Annie S Wu","year":"2011","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eccles et al., 2019] Tom Eccles, Edward Hughes, János Kramár, Steven Wheelwright, and Joel Z Leibo. Learning reciprocity in complex sequential social dilemmas. arXiv preprint arXiv:1903.08082, 2019.","title":"Learning reciprocity in complex sequential social dilemmas","authors":"Eccles","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fokker, 1914] Adriaan Daniël Fokker. Die mittlere energie rotierender elektrischer dipole im strahlungsfeld. Annalen der Physik, 348(5):810-820, 1914.","title":"Die mittlere energie rotierender elektrischer dipole im strahlungsfeld","authors":"Adriaan Daniël, Fokker","year":"1914","venue":"Annalen der Physik","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gomes and Kowalczyk, 2009] Eduardo Rodrigues Gomes and Ryszard Kowalczyk. Dynamic analysis of multia- gent q-learning with ε-greedy exploration. In ICML, pages 369-376, 2009.","title":"Dynamic analysis of multiagent q-learning with ε-greedy exploration","authors":"Kowalczyk Gomes, Eduardo Rodrigues Gomes, Ryszard Kowalczyk","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodman, 1953] Leo A Goodman. Population growth of the sexes. Biometrics, 9(2):212-225, 1953.","title":"Population growth of the sexes","authors":"Leo A Goodman","year":"1953","venue":"Biometrics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hao and Leung, 2013] Jianye Hao and Ho-fung Leung. The dynamics of reinforcement social learning in cooperative multiagent systems. In IJCAI, volume 13, pages 184-190, 2013.","title":"The dynamics of reinforcement social learning in cooperative multiagent systems","authors":"Leung Hao, Jianye Hao, Ho-Fung Leung","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/37"},{"position":10,"raw":"[Hu and Leung, 2017] Shuyue Hu and Ho-fung Leung. Achieving coordination in multi-agent systems by stable local conventions under community networks. In IJCAI, pages 4731-4737, 2017.","title":"Achieving coordination in multi-agent systems by stable local conventions under community networks","authors":"Leung Hu, Shuyue Hu, Ho-Fung Leung","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/659"},{"position":11,"raw":"[Hu et al., 2019] Shuyue Hu, Chin-wing Leung, and Ho- fung Leung. Modelling the dynamics of multiagent q- learning in repeated symmetric games: a mean field the- oretic approach. In NIPS, pages 12125-12135, 2019.","title":"Modelling the dynamics of multiagent qlearning in repeated symmetric games: a mean field theoretic approach","authors":"Hu","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Keshvadi and Faghih, 2016] Sina Keshvadi and Behnam Faghih. A multi-agent based load balancing system in iaas cloud environment. International Robotics & Automation Journal, 1(1):1-6, 2016.","title":"Sina Keshvadi and Behnam Faghih. A multi-agent based load balancing system in iaas cloud environment","authors":"Faghih Keshvadi","year":"2016","venue":"International Robotics & Automation Journal","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kianercy and Galstyan, 2012] Ardeshir Kianercy and Aram Galstyan. Dynamics of boltzmann q learning in two-player two-action games. Physical Review E, 85(4):041145, 2012.","title":"Ardeshir Kianercy and Aram Galstyan. Dynamics of boltzmann q learning in two-player two-action games","authors":"Galstyan Kianercy","year":"2012","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lanctot et al., 2017] Marc Lanctot, Vinicius Zambaldi, Au- drunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game- theoretic approach to multiagent reinforcement learning. In NIPS, pages 4190-4203, 2017.","title":"A unified gametheoretic approach to multiagent reinforcement learning","authors":"Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, Thore Graepel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Leibo et al., 2017] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In AAMAS, pages 464-473, 2017.","title":"Multi-agent reinforcement learning in sequential social dilemmas","authors":"Vinicius Joel Z Leibo, Marc Zambaldi, Janusz Lanctot, Thore Marecki, Graepel","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lesser et al., 2003] Victor Lesser, Charles L Ortiz Jr, and Milind Tambe. Distributed sensor networks: A multiagent perspective, volume 9. Springer Science & Business Me- dia, 2003.","title":"Distributed sensor networks: A multiagent perspective","authors":"Victor Lesser, Charles L Ortiz, Milind Tambe","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McKee et al., 2020] Kevin R McKee, Ian Gemp, Brian McWilliams, Edgar A Duéñez-Guzmán, Edward Hughes, and Joel Z Leibo. Social diversity and social preferences in mixed-motive reinforcement learning. arXiv preprint arXiv:2002.02325, 2020.","title":"Social diversity and social preferences in mixed-motive reinforcement learning","authors":"Ian Kevin R Mckee, Brian Gemp, Edgar A Mcwilliams, Edward Duéñez-Guzmán, Joel Z Hughes, Leibo","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Planck, 1917] M. Planck. Über einen Satz der statistischen Dynamik und seine Erweiterung in der Quantentheorie. Sitzungsberichte der Königlich-Preussischen Akademie der Wissenschaften zu Berlin. Reimer, 1917.","title":"Über einen Satz der statistischen Dynamik und seine Erweiterung in der Quantentheorie","authors":"M Planck","year":"1917","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sen and Airiau, 2007] Sandip Sen and Stéphane Airiau. Emergence of norms through social learning. In IJCAI, volume 1507, page 1512, 2007.","title":"Emergence of norms through social learning","authors":"Sandip Sen, Stéphane Airiau","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/243"},{"position":20,"raw":"[Tuyls et al., 2003] Karl Tuyls, Katja Verbeeck, and Tom Lenaerts. A selection-mutation model for q-learning in multi-agent systems. In AAMAS, pages 693-700, 2003.","title":"A selection-mutation model for q-learning in multi-agent systems","authors":"Tuyls","year":"2003","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tuyls et al., 2006] Karl Tuyls, Pieter Jan'T Hoen, and Bram Vanschoenwinkel. An evolutionary dynamical analy- sis of multi-agent learning in iterated games. AAMAS, 12(1):115-153, 2006.","title":"An evolutionary dynamical analysis of multi-agent learning in iterated games","authors":"Tuyls","year":"2006","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021] Jinghui Wang, Wei Lv, Yajuan Jiang, Shuangshuang Qin, and Jiawei Li. A multi-agent based cellular automata model for intersection traffic control simulation. Physica A: Statistical Mechanics and its Ap- plications, 584:126356, 2021.","title":"A multi-agent based cellular automata model for intersection traffic control simulation","authors":"Jinghui Wang, Wei Lv, Yajuan Jiang, Shuangshuang Qin, Jiawei Li","year":"2021","venue":"Physica A: Statistical Mechanics and its Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weiss, 1907] Pierre Weiss. L'hypothèse du champ moléculaire et la propriété ferromagnétique. J. Phys. Theor. Appl., 6(1):661-690, 1907.","title":"L'hypothèse du champ moléculaire et la propriété ferromagnétique","authors":"Pierre Weiss","year":"1907","venue":"J. Phys. Theor. Appl","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Whittle, 1957] Peter Whittle. On the use of the normal ap- proximation in the treatment of stochastic processes. Jour- nal of the Royal Statistical Society: Series B (Methodolog- ical), 19(2):268-281, 1957.","title":"On the use of the normal approximation in the treatment of stochastic processes","authors":"Peter Whittle, Whittle","year":"1957","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wunder et al., 2010] Michael Wunder, Michael L Littman, and Monica Babes. Classes of multiagent q-learning dy- namics with epsilon-greedy exploration. In ICML, pages 1167-1174. Citeseer, 2010.","title":"Classes of multiagent q-learning dynamics with epsilon-greedy exploration","authors":"Michael Wunder, Monica Michael L Littman, Babes","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wyns and Du Toit, 2017] Maarten Wyns and Jacques Du Toit. A finite volume-alternating direction implicit approach for the calibration of stochastic local volatility models. International Journal of Computer Mathematics, 94(11):2239-2267, 2017.","title":"A finite volume-alternating direction implicit approach for the calibration of stochastic local volatility models","authors":"Du Wyns, Maarten Toit, Jacques Wyns, Toit Du","year":"2017","venue":"International Journal of Computer Mathematics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2013] Chao Yu, Minjie Zhang, Fenghui Ren, and Xudong Luo. Emergence of social norms through col- lective learning in networked agent societies. In AAMAS, pages 475-482, 2013.","title":"Emergence of social norms through collective learning in networked agent societies","authors":"Chao Yu, Minjie Zhang, Fenghui Ren, Xudong Luo","year":"2013","venue":"AAMAS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}