{"type":"ScholarlyArticle","title":"Modelling the Dynamics of Regret Minimization in Large Agent Populations: a Master Equation Approach","authors":[{"name":"Zhen Wang","slug":"zhen-wang","given":"Zhen","family":"Wang","affiliation":"School of Cybersecurity , Northwestern Polytechnical University; School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"},{"name":"Chunjiang Mu","slug":"chunjiang-mu","given":"Chunjiang","family":"Mu","affiliation":"School of Cybersecurity , Northwestern Polytechnical University; School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"},{"name":"Shuyue Hu","slug":"shuyue-hu","given":"Shuyue","family":"Hu","affiliation":"Shanghai Artificial Intelligence Laboratory"},{"name":"Chen Chu","slug":"chen-chu","given":"Chen","family":"Chu","affiliation":"School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University; School of Statistics and Mathematics , Yunnan University of Finance and Economics"},{"name":"Xuelong Li","slug":"xuelong-li","given":"Xuelong","family":"Li","affiliation":"School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"}],"abstract":"Understanding the learning dynamics in multiagent systems is an important and challenging task. Past research on multi-agent learning mostly focuses on two-agent settings. In this paper, we consider the scenario in which a population of infinitely many agents apply regret minimization in repeated symmetric games. We propose a new formal model based on the master equation approach in statistical physics to describe the evolutionary dynamics in the agent population. Our model takes the form of a partial differential equation, which describes how the probability distribution of regret evolves over time. Through experiments, we show that our theoretical results are consistent with the agent-based simulation results.","doi":"10.24963/ijcai.2022/76","url":"https://ijcai.org/proceedings/2022/76","pdf":"https://ijcai.org/proceedings/2022/0076.pdf","markdown":"https://ijcai.org/proceedings/2022/76.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/76","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":76,"pages":{"first":534,"last":540},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986751","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984894","length_sec":303,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Barbosa and Belo, 2010] Rui Pedro Barbosa and Orlando Belo. Multi-agent forex trading system. In Agent and multi-agent technology for internet and enterprise sys- tems, pages 91-118. Springer, 2010.","title":"Multi-agent forex trading system. In Agent and multi-agent technology for internet and enterprise systems","authors":"Pedro Rui, Orlando Barbosa, Belo","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bloembergen et al., 2015] Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dy- namics of multi-agent learning: A survey. Journal of Arti- ficial Intelligence Research, 53:659-697, 2015.","title":"Evolutionary dynamics of multi-agent learning: A survey","authors":"Daan Bloembergen, Karl Tuyls, Daniel Hennes, Michael Kaisers","year":"2015","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blum and Monsour, 2007] Avrim Blum and Yishay Mon- sour. Learning, regret minimization, and equilibria. Al- gorithmic Game Theory, pages 79-100, 2007.","title":"Learning, regret minimization, and equilibria. Algorithmic Game Theory","authors":"Avrim Blum, Yishay Monsour","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Börgers and Sarin, 1997] Tilman Börgers and Rajiv Sarin. Learning through reinforcement and replicator dynamics. Journal of economic theory, 77(1):1-14, 1997.","title":"Learning through reinforcement and replicator dynamics","authors":"Sarin Börgers, Tilman Börgers, Rajiv Sarin","year":"1997","venue":"Journal of economic theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown and Sandholm, 2019] Noam Brown and Tuomas Sandholm. Solving imperfect-information games via dis- counted regret minimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1829-1836, 2019.","title":"Solving imperfect-information games via discounted regret minimization","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown et al., 2019] Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International conference on machine learn- ing, pages 793-802. PMLR, 2019.","title":"Deep counterfactual regret minimization","authors":"Brown","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hart and Mas-Colell, 2000] Sergiu Hart and Andreu Mas- Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica, 68(5):1127-1150, 2000.","title":"A simple adaptive procedure leading to correlated equilibrium","authors":"Sergiu Hart, Andreu Mas-Colell","year":"2000","venue":"Econometrica","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2019] Shuyue Hu, Chin-wing Leung, and Ho- fung Leung. Modelling the dynamics of multiagent q- learning in repeated symmetric games: a mean field the- oretic approach. Advances in Neural Information Process- ing Systems, 32:12125-12135, 2019.","title":"Modelling the dynamics of multiagent qlearning in repeated symmetric games: a mean field theoretic approach","authors":"Hu","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2020] Shuyue Hu, Chin-Wing Leung, Ho-fung Leung, and Harold Soh. The evolutionary dynamics of independent learning agents in population games. arXiv preprint arXiv:2006.16068, 2020.","title":"The evolutionary dynamics of independent learning agents in population games","authors":"Hu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2019] Max Jaderberg, Wojciech M Czar- necki, Iain Dunning, Luke Marris, Guy Lever, Anto- nio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364(6443):859- 865, 2019.","title":"Human-level performance in 3d multiplayer games with populationbased reinforcement learning","authors":"Jaderberg","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaisers and Tuyls, 2010] Michael Kaisers and Karl Tuyls. Frequency adjusted multi-agent q-learning. In Proceed- ings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1, pages 309-316, 2010.","title":"Frequency adjusted multi-agent q-learning","authors":"Tuyls Kaisers, Michael Kaisers, Karl Tuyls","year":"2010","venue":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kaisers et al., 2012] Michael Kaisers, Daan Bloembergen, and Karl Tuyls. A common gradient in multi-agent rein- forcement learning. In AAMAS, pages 1393-1394, 2012.","title":"A common gradient in multi-agent reinforcement learning","authors":"Michael Kaisers, Daan Bloembergen, Karl Tuyls","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Klos et al., 2010] Tomas Klos, Gerrit Jan Van Ahee, and Karl Tuyls. Evolutionary dynamics of regret minimiza- tion. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pages 82-96. Springer, 2010.","title":"Evolutionary dynamics of regret minimization","authors":"Klos","year":"2010","venue":"Joint European Conference on Machine Learn-ing and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lahkar and Seymour, 2013] Ratul Lahkar and Robert M Seymour. Reinforcement learning in population games. Games and Economic Behavior, 80:10-38, 2013.","title":"Reinforcement learning in population games","authors":"Seymour Lahkar","year":"2013","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mandel and Wolf, 1995] Leonard Mandel and Emil Wolf. Optical coherence and quantum optics. Cambridge uni- versity press, 1995.","title":"Optical coherence and quantum optics","authors":"Wolf ; Mandel, Leonard Mandel, Emil Wolf","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ota, 2006] Jun Ota. Multi-agent robot systems as dis- tributed autonomous systems. Advanced engineering in- formatics, 20(1):59-70, 2006.","title":"Jun Ota. Multi-agent robot systems as distributed autonomous systems","authors":"Ota","year":"2006","venue":"Advanced engineering informatics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tuyls et al., 2003] Karl Tuyls, Katja Verbeeck, and Tom Lenaerts. A selection-mutation model for q-learning in multi-agent systems. In Proceedings of the second inter- national joint conference on Autonomous agents and mul- tiagent systems, pages 693-700, 2003.","title":"A selection-mutation model for q-learning in multi-agent systems","authors":"Tuyls","year":"2003","venue":"Proceedings of the second international joint conference on Autonomous agents and multiagent systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2020] Jiachen Yang, Jipeng Zhang, and Huihui Wang. Urban traffic control in software defined internet of things via a multi-agent deep reinforcement learning ap- proach. IEEE Transactions on Intelligent Transportation Systems, 22(6):3742-3754, 2020.","title":"Urban traffic control in software defined internet of things via a multi-agent deep reinforcement learning approach","authors":"Yang","year":"2020","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}