{"type":"ScholarlyArticle","title":"A Formal Model for Multiagent Q-Learning Dynamics on Regular Graphs","authors":[{"name":"Chen Chu","slug":"chen-chu","given":"Chen","family":"Chu","affiliation":"School of Statistics and Mathematics , Yunnan University of Finance and Economics; School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"},{"name":"Yong Li","slug":"yong-li","given":"Yong","family":"Li","affiliation":"School of Software , Yunnan University"},{"name":"Jinzhuo Liu","slug":"jinzhuo-liu","given":"Jinzhuo","family":"Liu","affiliation":"School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University; School of Software , Yunnan University"},{"name":"Shuyue Hu","slug":"shuyue-hu","given":"Shuyue","family":"Hu","affiliation":"Shanghai Artificial Intelligence Laboratory"},{"name":"Xuelong Li","slug":"xuelong-li","given":"Xuelong","family":"Li","affiliation":"School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"},{"name":"Zhen Wang","slug":"zhen-wang","given":"Zhen","family":"Wang","affiliation":"School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University; School of Cybersecurity , Northwestern Polytechnical University"}],"abstract":"Modeling the dynamics of multi-agent learning has long been an important research topic. The focus of previous research has been either on 2-agent settings or well-mixed infinitely large agent populations. In this paper, we consider the scenario where n Q-learning agents locate on regular graphs, such that agents can only interact with their neighbors. We examine the local interactions between individuals and their neighbors, and derive a formal model to capture the Q-value dynamics of the entire population. Through comparisons with agent-based simulations on different types of regular graphs, we show that our model describes the agent learning dynamics in an exact manner.","doi":"10.24963/ijcai.2022/28","url":"https://ijcai.org/proceedings/2022/28","pdf":"https://ijcai.org/proceedings/2022/0028.pdf","markdown":"https://ijcai.org/proceedings/2022/28.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/28","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":28,"pages":{"first":194,"last":200},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Agent Societies","Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987095","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985238","length_sec":645,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bailey and Piliouras, 2019] James P Bailey and Georgios Piliouras. Multi-agent learning in network zero-sum games is a hamiltonian system. In Proceedings of the 18th International Conference on Autonomous Agents and Mul- tiAgent Systems, pages 233-241. International Foundation for Autonomous Agents and Multiagent Systems, 2019.","title":"Multi-agent learning in network zero-sum games is a hamiltonian system","authors":"P James, Georgios Bailey, Piliouras","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and Mul-tiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bloembergen et al., 2015] Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dy- namics of multi-agent learning: A survey. Journal of Arti- ficial Intelligence Research, 53:659-697, 2015.","title":"Evolutionary dynamics of multi-agent learning: A survey","authors":"Daan Bloembergen, Karl Tuyls, Daniel Hennes, Michael Kaisers","year":"2015","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown and Sandholm, 2019] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456):885-890, 2019.","title":"Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker","authors":"Sandholm Brown","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Delgado, 2002] Jordi Delgado. Emergence of social con- ventions in complex networks. Artificial intelligence, 141(1-2):171-185, 2002.","title":"Emergence of social conventions in complex networks","authors":"Jordi Delgado, Delgado","year":"2002","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Galstyan, 2013] Aram Galstyan. Continuous strategy repli- cator dynamics for multi-agent q-learning. Autonomous agents and multi-agent systems, 26(1):37-53, 2013.","title":"Aram Galstyan. Continuous strategy replicator dynamics for multi-agent q-learning","authors":"Galstyan","year":"2013","venue":"Autonomous agents and multi-agent systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hennes et al., 2009] Daniel Hennes, Karl Tuyls, and Matthias Rauterberg. State-coupled replicator dynamics. In Proceedings of The 8th International Conference on Au- tonomous Agents and Multiagent Systems-Volume 2, pages 789-796, 2009.","title":"State-coupled replicator dynamics","authors":"Daniel Hennes, Karl Tuyls, Matthias Rauterberg","year":"2009","venue":"Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu and Leung, 2017] Shuyue Hu and Ho-fung Leung. Achieving coordination in multi-agent systems by stable local conventions under community networks. In Ijcai, pages 4731-4737, 2017.","title":"Achieving coordination in multi-agent systems by stable local conventions under community networks","authors":"Leung Hu, Shuyue Hu, Ho-Fung Leung","year":"2017","venue":"Ijcai","doi":null,"url":null,"internal_url":"/proceedings/2017/659"},{"position":8,"raw":"[Hu et al., 2019] Shuyue Hu, Chin-wing Leung, and Ho- fung Leung. Modelling the dynamics of multiagent q- learning in repeated symmetric games: a mean field the- oretic approach. Advances in Neural Information Process- ing Systems, 32:12125-12135, 2019.","title":"Modelling the dynamics of multiagent qlearning in repeated symmetric games: a mean field theoretic approach","authors":"Hu","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2018] Junqi Jin, Chengru Song, Han Li, Kun Gai, Jun Wang, and Weinan Zhang. Real-time bidding with multi-agent reinforcement learning in display advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 2193- 2201, 2018.","title":"Real-time bidding with multi-agent reinforcement learning in display advertising","authors":"Jin","year":"2018","venue":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kaisers and Tuyls, 2010] Michael Kaisers and Karl Tuyls. Frequency adjusted multi-agent q-learning. In Proceed- ings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1, pages 309-316, 2010.","title":"Frequency adjusted multi-agent q-learning","authors":"Tuyls Kaisers, Michael Kaisers, Karl Tuyls","year":"2010","venue":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klos et al., 2010] Tomas Klos, Gerrit Jan Van Ahee, and Karl Tuyls. Evolutionary dynamics of regret minimiza- tion. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pages 82-96. Springer, 2010.","title":"Evolutionary dynamics of regret minimization","authors":"Klos","year":"2010","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2021a] Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Eslami, Daniel Hennes, Wojciech M Czar- necki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdol- maleki, et al. From motor control to team play in simulated humanoid football. arXiv preprint arXiv:2105.12196, 2021.","title":"From motor control to team play in simulated humanoid football","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021b] Yiwei Liu, Jiamou Liu, Kaibin Wan, Zhan Qin, Zijian Zhang, Bakhadyr Khoussainov, and Liehuang Zhu. From local to global norm emergence: Dissolving self-reinforcing substructures with incremental social in- struments. In International Conference on Machine Learn- ing, pages 6871-6881. PMLR, 2021.","title":"From local to global norm emergence: Dissolving self-reinforcing substructures with incremental social instruments","authors":"Liu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Panait et al., 2008] Liviu Panait, Karl Tuyls, and Sean Luke. Theoretical advantages of lenient learners: An evolution- ary game theoretic perspective. The Journal of Machine Learning Research, 9:423-457, 2008.","title":"Theoretical advantages of lenient learners: An evolutionary game theoretic perspective","authors":"Liviu Panait, Karl Tuyls, Sean Luke","year":"2008","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sen and Sen, 2009] Onkur Sen and Sandip Sen. Effects of social network topology and options on norm emer- gence. In International Workshop on Coordination, Orga- nizations, Institutions, and Norms in Agent Systems, pages 211-222. Springer, 2009.","title":"Onkur Sen and Sandip Sen. Effects of social network topology and options on norm emergence","authors":"Sen Sen","year":"2009","venue":"International Workshop on Coordination, Organizations, Institutions, and Norms in Agent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tuyls and Parsons, 2007] Karl Tuyls and Simon Parsons. What evolutionary game theory tells us about multiagent learning. Artificial Intelligence, 171(7):406-416, 2007.","title":"What evolutionary game theory tells us about multiagent learning","authors":"Parsons Tuyls, Karl Tuyls, Simon Parsons","year":"2007","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tuyls et al., 2003] Karl Tuyls, Katja Verbeeck, and Tom Lenaerts. A selection-mutation model for q-learning in multi-agent systems. In Proceedings of the second inter- national joint conference on Autonomous agents and mul- tiagent systems, pages 693-700, 2003.","title":"A selection-mutation model for q-learning in multi-agent systems","authors":"Tuyls","year":"2003","venue":"Proceedings of the second international joint conference on Autonomous agents and multiagent systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Villatoro et al., 2011] Daniel Villatoro, Jordi Sabater-Mir, and Sandip Sen. Social instruments for robust convention emergence. In Twenty-Second International Joint Confer- ence on Artificial Intelligence, 2011.","title":"Social instruments for robust convention emergence","authors":"Daniel Villatoro, Jordi Sabater-Mir, Sandip Sen","year":"2011","venue":"Twenty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2011/78"},{"position":19,"raw":"[Wang et al., 2018] Yixi Wang, Wenhuan Lu, Jianye Hao, Jianguo Wei, and Ho-fung Leung. Efficient convention emergence through decoupled reinforcement social learn- ing with teacher-student mechanism. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 795-803, 2018.","title":"Efficient convention emergence through decoupled reinforcement social learning with teacher-student mechanism","authors":"Wang","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2018] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentral- ized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learn- ing, pages 5872-5881. PMLR, 2018.","title":"Fully decentralized multi-agent reinforcement learning with networked agents","authors":"Zhang","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2021] Kaiqing Zhang, Zhuoran Yang, and Tamer Bas ¸ar. Multi-agent reinforcement learning: A se- lective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.","title":"Multi-agent reinforcement learning: A selective overview of theories and algorithms","authors":"Zhang","year":"2021","venue":"Handbook of Reinforcement Learning and Control","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2019] Ming Zhou, Jiarui Jin, Weinan Zhang, Zhiwei Qin, Yan Jiao, Chenxi Wang, Guobin Wu, Yong Yu, and Jieping Ye. Multi-agent reinforcement learning for order-dispatching via order-vehicle distribution match- ing. In Proceedings of the 28th ACM International Confer- ence on Information and Knowledge Management, pages 2645-2653, 2019.","title":"Multi-agent reinforcement learning for order-dispatching via order-vehicle distribution matching","authors":"Ming Zhou, Jiarui Jin, Weinan Zhang, Zhiwei Qin, Yan Jiao, Chenxi Wang, Guobin Wu, Yong Yu, Jieping Ye","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}