{"type":"ScholarlyArticle","title":"M2RL: A Multi-player Multi-agent Reinforcement Learning Framework for Complex Games","authors":[{"name":"Tongtong Yu","slug":"tongtong-yu","given":"Tongtong","family":"Yu","affiliation":"Institute of Automation , Chinese Academy of Sciences , China"},{"name":"Chenghua He","slug":"chenghua-he","given":"Chenghua","family":"He","affiliation":"Institute of Automation , Chinese Academy of Sciences , China"},{"name":"Qiyue Yin","slug":"qiyue-yin","given":"Qiyue","family":"Yin","affiliation":"Institute of Automation , Chinese Academy of Sciences , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , China"}],"abstract":"Distributed deep reinforcement learning (DDRL) has gained increasing attention due to the emerging requirements for addressing complex games like Go and StarCraft. However, how to effectively and stably train bots with asynchronous and heterogeneous agents cooperation and competition for multiple players under multiple machines (with multiple CPUs and GPUs) using DDRL is still an open problem. We propose and open M2RL, a Multi-player and Multi-agent Reinforcement Learning framework, to make training bots for complex games an easy-to-use warehouse. Experiments involving training a two-player multi-agent Wargame AI, and a sixteen-player multi-agent community game Neural MMO AI, demonstrate the effectiveness of the proposed framework by winning a silver award and beating high-level AI bots designed by professional players.","doi":"10.24963/ijcai.2024/1046","url":"https://ijcai.org/proceedings/2024/1046","pdf":"https://ijcai.org/proceedings/2024/1046.pdf","markdown":"https://ijcai.org/proceedings/2024/1046.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1046","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1046,"pages":{"first":8847,"last":8850},"keywords":["Agent-based and Multi-agent Systems: MAS: Applications","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Uncertainty in AI: UAI: Sequential decision making","Agent-based and Multi-agent Systems: MAS: Engineering methods, platforms, languages and tools"],"videos":[],"references":[{"position":1,"raw":"[Accinelli and Carrera, 2011] Elvio Accinelli and Edgar J Sánchez Carrera. Evolutionarily stable strategies and replicator dynamics in asymmetric two-population games. In Dynamics, Games and Science I: DYNA 2008, in Honor of Maurício Peixoto and David Rand, University of Minho, Braga, Portugal, September 8-12, 2008, pages 25-35. Springer, 2011.","title":"Evolutionarily stable strategies and replicator dynamics in asymmetric two-population games","authors":"Elvio Accinelli, Edgar J Sánchez Carrera","year":"2008","venue":"Dynamics, Games and Science I: DYNA 2008","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berner et al., 2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforce- ment learning. arXiv preprint arXiv:1912.06680, 2019.","title":"Dota 2 with large scale deep reinforcement learning","authors":"Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Espeholt et al., 2019] Lasse Espeholt, Raphaël Marinier, Pi- otr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central in- ference. arXiv preprint arXiv:1910.06591, 2019.","title":"Seed rl: Scalable and efficient deep-rl with accelerated central inference","authors":"Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, Marcin Michalski","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Heinrich and Silver, 2016] Johannes Heinrich and David Silver. Deep reinforcement learning from self- play in imperfect-information games. arXiv preprint arXiv:1603.01121, 2016.","title":"Deep reinforcement learning from selfplay in imperfect-information games","authors":"Silver Heinrich, Johannes Heinrich, David Silver","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Herbrich et al., 2006] Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill™: a bayesian skill rating sys- tem. Advances in neural information processing systems, 19, 2006.","title":"Trueskill™: a bayesian skill rating system","authors":"Ralf Herbrich, Tom Minka, Thore Graepel","year":"2006","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hoffman et al., 2020] Matthew W Hoffman, Bobak Shahri- ari, John Aslanides, Gabriel Barth-Maron, Nikola Mom- chev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020.","title":"Acme: A research framework for distributed reinforcement learning","authors":"Bobak Matthew W Hoffman, John Shahriari, Gabriel Aslanides, Nikola Barth-Maron, Danila Momchev, Piotr Sinopalnikov, Sabela Stańczyk, Anton Ramos, Damien Raichuk, Vincent","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Malekmohammadi, 2021] Saber Malekmohammadi. Ray: A distributed framework for emerging ai applications. 2021.","title":"Saber Malekmohammadi. Ray: A distributed framework for emerging ai applications","authors":"Malekmohammadi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Pretorius et al., 2021] Arnu Pretorius, Kale-ab Tessera, An- dries P Smit, Claude Formanek, St John Grimbly, Kevin Eloff, Siphelele Danisa, Lawrence Francis, Jonathan Shock, Herman Kamper, et al. Mava: A research frame- work for distributed multi-agent reinforcement learning. arXiv preprint arXiv:2107.01460, 2021.","title":"Mava: A research framework for distributed multi-agent reinforcement learning","authors":"Pretorius","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Suarez et al., 2019] Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.00784, 2019.","title":"Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents","authors":"Joseph Suarez, Yilun Du, Phillip Isola, Igor Mordatch","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Vinyals","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ye et al., 2020] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex con- trol in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 6672-6679, 2020.","title":"Mastering complex control in moba games with deep reinforcement learning","authors":"Ye","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Yin et al., 2023] Qiyue Yin, Meijing Zhao, Wancheng Ni, Junge Zhang, and Kaiqi Huang. Intelligent decision mak- ing technology and challenge of wargame. Acta Automat- ica Sinica, 49(5):913-928, 2023.","title":"Intelligent decision making technology and challenge of wargame","authors":"Qiyue Yin, Meijing Zhao, Wancheng Ni, Junge Zhang, Kaiqi Huang","year":"2023","venue":"Acta Automatica Sinica","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Yin et al., 2024] Qiyue Yin, Tongtong Yu, Shengqi Shen, Jun Yang, Meijing Zhao, Kaiqi Huang, Bin Liang, and Liang Wang. Distributed deep reinforcement learning: A survey and a multi-player multi-agent learning toolbox. Machine Intelligence Research, 21:411-430, 2024.","title":"Distributed deep reinforcement learning: A survey and a multi-player multi-agent learning toolbox","authors":"Qiyue Yin, Tongtong Yu, Shengqi Shen, Jun Yang, Meijing Zhao, Kaiqi Huang, Bin Liang, Liang Wang","year":"2024","venue":"Machine Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Zhou et al., 2023] Ming Zhou, Ziyu Wan, Hanjing Wang, Muning Wen, Runzhe Wu, Ying Wen, Yaodong Yang, Yong Yu, Jun Wang, and Weinan Zhang. Malib: A paral- lel framework for population-based multi-agent reinforce- ment learning. J. Mach. Learn. Res., 24:150-1, 2023.","title":"Malib: A parallel framework for population-based multi-agent reinforcement learning","authors":"Ming Zhou, Ziyu Wan, Hanjing Wang, Muning Wen, Runzhe Wu, Ying Wen, Yaodong Yang, Yong Yu, Jun Wang, Weinan Zhang","year":"2023","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}