{"type":"ScholarlyArticle","title":"Towards a Better Understanding of Learning with Multiagent Teams","authors":[{"name":"David Radke","slug":"david-radke","given":"David","family":"Radke","affiliation":"School of Computer Science , University of Waterloo"},{"name":"Kate Larson","slug":"kate-larson","given":"Kate","family":"Larson","affiliation":"School of Computer Science , University of Waterloo"},{"name":"Tim Brecht","slug":"tim-brecht","given":"Tim","family":"Brecht","affiliation":"School of Computer Science , University of Waterloo"},{"name":"Kyle Tilbury","slug":"kyle-tilbury","given":"Kyle","family":"Tilbury","affiliation":"School of Computer Science , University of Waterloo"}],"abstract":"While it has long been recognized that a team of individual learning agents can be greater than the sum of its parts, recent work has shown that larger teams are not necessarily more effective than smaller ones. In this paper, we study why and under which conditions certain team structures promote effective learning for a population of individual learning agents. We show that, depending on the environment, some team structures help agents learn to specialize into specific roles, resulting in more favorable global results. However, large teams create credit assignment challenges that reduce coordination, leading to large teams performing poorly compared to smaller ones. We support our conclusions with both theoretical analysis and empirical results.","doi":"10.24963/ijcai.2023/31","url":"https://ijcai.org/proceedings/2023/31","pdf":"https://ijcai.org/proceedings/2023/0031.pdf","markdown":"https://ijcai.org/proceedings/2023/31.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/31","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":31,"pages":{"first":271,"last":279},"keywords":["Agent-based and Multi-agent Systems: MAS: Agent-based simulation and emergence","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Arjona-Medina et al., 2019] Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. NeurIPS, 32, 2019.","title":"Rudder: Return decomposition for delayed rewards","authors":"Michael Jose A Arjona-Medina, Michael Gillhofer, Thomas Widrich, Johannes Unterthiner, Sepp Brandstetter, Hochreiter","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arumugam et al., 2020] Dilip Arumugam, Peter Hender- son, and Pierre-Luc Bacon. An information-theoretic perspective on credit assignment in reinforcement learn- ing. Workshop on Biological and Artificial Reinforcement Learning (NeurIPS 2020), 2020.","title":"An information-theoretic perspective on credit assignment in reinforcement learning","authors":"Dilip Arumugam, Peter Henderson, Pierre-Luc Bacon","year":"2020","venue":"Workshop on Biological and Artificial Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baker et al., 2019] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent au- tocurricula. ICLR, 2019.","title":"Emergent tool use from multi-agent autocurricula","authors":"Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob Mcgrew, Igor Mordatch","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dafoe et al., 2020] Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in coop- erative AI. arXiv preprint arXiv:2012.08630, 2020.","title":"Open problems in cooperative AI","authors":"Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R Mckee, Joel Z Leibo, Kate Larson, Thore Graepel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2021] Heng Dong, Tonghan Wang, Jiayuan Liu, Chi Han, and Chongjie Zhang. Birds of a feather flock together: A close look at cooperation emergence via multi-agent rl. arXiv preprint arXiv:2104.11455, 2021.","title":"Birds of a feather flock together: A close look at cooperation emergence via multi-agent rl","authors":"Heng Dong, Tonghan Wang, Jiayuan Liu, Chi Han, Chongjie Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Durugkar et al., 2020] Ishan Durugkar, E. Liebman, and P. Stone. Balancing individual preferences and shared ob- jectives in multiagent reinforcement learning. In IJCAI, 2020.","title":"Balancing individual preferences and shared objectives in multiagent reinforcement learning","authors":"E Ishan Durugkar, P Liebman, Stone","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/347"},{"position":7,"raw":"[Gemp et al., 2022] Ian Gemp, Kevin R McKee, Richard Ev- erett, Edgar A Duéñez-Guzmán, Yoram Bachrach, David Balduzzi, and Andrea Tacchetti. D3C: Reducing the price of anarchy in multi-agent learning. AAMAS, 2022.","title":"D3C: Reducing the price of anarchy in multi-agent learning","authors":"Ian Gemp, Kevin R Mckee, Richard Everett, Edgar A Duéñez-Guzmán, Yoram Bachrach, David Balduzzi, Andrea Tacchetti","year":"2022","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gürtler et al., 2021] Nico Gürtler, Dieter Büchler, and Georg Martius. Hierarchical reinforcement learning with timed subgoals. NeurIPS, 34:21732-21743, 2021.","title":"Hierarchical reinforcement learning with timed subgoals","authors":"Nico Gürtler, Dieter Büchler, Georg Martius","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Konidaris and Barto, 2007] George Dimitri Konidaris and Andrew G Barto. Building portable options: Skill trans- fer in reinforcement learning. In IJCAI, volume 7, pages 895-900, 2007.","title":"Building portable options: Skill transfer in reinforcement learning","authors":"Barto ; Konidaris, Dimitri George, Andrew G Konidaris, Barto","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/144"},{"position":10,"raw":"[Leibo et al., 2017] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. AA- MAS, 2017.","title":"Multi-agent reinforcement learning in sequential social dilemmas","authors":"Vinicius Joel Z Leibo, Marc Zambaldi, Janusz Lanctot, Thore Marecki, Graepel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mathieu et al., 2001] John E. Mathieu, Michelle A. Marks, and Stephen J. Zaccaro. Multiteam systems. Handbook of Industrial, Work, and Ordanizational Psychology, 2, 2001.","title":"Multiteam systems","authors":"John E Mathieu, Michelle A Marks, Stephen J Zaccaro","year":"2001","venue":"Handbook of Industrial, Work, and Ordanizational Psychology","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Matignon et al., 2007] Laëtitia Matignon, Guillaume J Lau- rent, and Nadine Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in co- operative multi-agent teams. In IROS, pages 64-69. IEEE, 2007.","title":"Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams","authors":"Laëtitia Matignon, Guillaume J Laurent, Nadine Le Fort-Piat","year":"2007","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McKee et al., 2020] Kevin R. McKee, Ian Gemp, Brian McWilliams, Edgar A. Duéñez-Guzmán, Edward Hughes, and Joel Z. Leibo. Social diversity and social preferences in mixed-motive reinforcement learning. AAMAS, 2020.","title":"Social diversity and social preferences in mixed-motive reinforcement learning","authors":"Kevin R Mckee, Ian Gemp, Brian Mcwilliams, Edgar A Duéñez-Guzmán, Edward Hughes, Joel Z Leibo","year":"2020","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nachum et al., 2018] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierar- chical reinforcement learning. NeurIPS, 31, 2018.","title":"Data-efficient hierarchical reinforcement learning","authors":"Ofir Nachum, Shane Shixiang, Honglak Gu, Sergey Lee, Levine","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, vol- ume 99, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Palmer et al., 2019] Gregory Palmer, Rahul Savani, and Karl Tuyls. Negative update intervals in deep multi-agent reinforcement learning. AAMAS, 2019.","title":"Negative update intervals in deep multi-agent reinforcement learning","authors":"Gregory Palmer, Rahul Savani, Karl Tuyls","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Phan et al., 2021] Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann, Thomas Gabor, and Claudia Linnhoff- Popien. VAST: Value function factorization with variable agent sub-teams. NeurIPS, 34:24018-24032, 2021.","title":"VAST: Value function factorization with variable agent sub-teams","authors":"Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann, Thomas Gabor, Claudia Linnhoff-Popien","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pollack, 1986] Martha E Pollack. A model of plan infer- ence that distinguishes between the beliefs of actors and observers. New York, pages 207-214, 1986.","title":"A model of plan inference that distinguishes between the beliefs of actors and observers","authors":"Martha E Pollack","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pollack, 1990] Martha E. Pollack. Plans as complex mental attitudes. In Intentions in Communication, pages 77-103.","title":"Plans as complex mental attitudes","authors":"Martha E Pollack","year":null,"venue":"Intentions in Communication","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radke and Tilbury, 2023] David Radke and Kyle Tilbury. Learning to learn group alignment: A self-tuning credo framework with multiagent teams. ALA Workshop at AA- MAS, 2023.","title":"Learning to learn group alignment: A self-tuning credo framework with multiagent teams","authors":"Tilbury ; ] Radke, Radke David, Kyle Tilbury","year":"2023","venue":"ALA Workshop","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Radke et al., 2022] David Radke, Kate Larson, and Tim Brecht. Exploring the benefits of teams in multiagent learning. In IJCAI, 2022.","title":"Exploring the benefits of teams in multiagent learning","authors":"David Radke, Kate Larson, Tim Brecht","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/65"},{"position":22,"raw":"[Radke et al., 2023] David Radke, Kate Larson, and Tim Brecht. The importance of credo in multiagent learning. AAMAS, 2023.","title":"The importance of credo in multiagent learning","authors":"David Radke, Kate Larson, Tim Brecht","year":"2023","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rapoport, 1974] Anatol Rapoport. Prisoner's dilemma-recollections and observations. In Game Theory as a Theory of a Conflict Resolution, pages 17-34.","title":"Prisoner's dilemma-recollections and observations","authors":"Anatol Rapoport","year":null,"venue":"Game Theory as a Theory of a Conflict Resolution","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rashid et al., 2020] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Fo- erster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234- 7284, 2020.","title":"Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De, Gregory Witt, Jakob Farquhar, Shimon Foerster, Whiteson","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shannon, 1948] Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Jour- nal, 27(3):379-423, 1948.","title":"A mathematical theory of communication","authors":"Claude Elwood, Shannon","year":"1948","venue":"The Bell System Technical Journal","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stone et al., 2010] Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1504-1509, 2010.","title":"Ad hoc autonomous agent teams: Collaboration without pre-coordination","authors":"Peter Stone, Gal Kaminka, Sarit Kraus, Jeffrey Rosenschein","year":"2010","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Suarez et al., 2019] Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural MMO: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.00784, 2019.","title":"Neural MMO: A massively multiagent game environment for training and evaluating intelligent agents","authors":"Joseph Suarez, Yilun Du, Phillip Isola, Igor Mordatch","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tambe, 1997] Milind Tambe. Towards flexible teamwork. Journal of Artificial Intelligence Research, 7:83-124, 1997.","title":"Towards flexible teamwork","authors":"Milind Tambe","year":"1997","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Vinitsky et al., 2019] Eugene Vinitsky, Natasha Jaques, Joel Leibo, Antonio Castenada, and Edward Hughes. An open source implementation of sequential social dilemma games. https://github.com/eugenevinitsky/sequential social dilemma games/issues/182, 2019. Accessed: 2022- 07-01.","title":"An open source implementation of sequential social dilemma games","authors":"Eugene Vinitsky, Natasha Jaques, Joel Leibo, Antonio Castenada, Edward Hughes","year":"2019","venue":null,"doi":null,"url":"https://github.com/eugenevinitsky/sequentialsocialdilemmagames/issues/182","internal_url":null},{"position":34,"raw":"[Wang et al., 2021] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.","title":"A survey on curriculum learning","authors":"Xin Wang, Yudong Chen, Wenwu Zhu","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wiewiora et al., 2003] Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising re- inforcement learning agents. In ICML, pages 792-799, 2003.","title":"Principled methods for advising reinforcement learning agents","authors":"Eric Wiewiora, W Garrison, Charles Cottrell, Elkan","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang et al., 2020] Jiachen Yang, Ang Li, Mehrdad Fara- jtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning to incentivize other learning agents. NeurIPS, 33:15208-15219, 2020.","title":"Learning to incentivize other learning agents","authors":"Yang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}