{"type":"ScholarlyArticle","title":"Exploring the Benefits of Teams in Multiagent Learning","authors":[{"name":"David Radke","slug":"david-radke","given":"David","family":"Radke","affiliation":"School of Computer Science , University of Waterloo"},{"name":"Kate Larson","slug":"kate-larson","given":"Kate","family":"Larson","affiliation":"School of Computer Science , University of Waterloo"},{"name":"Tim Brecht","slug":"tim-brecht","given":"Tim","family":"Brecht","affiliation":"School of Computer Science , University of Waterloo"}],"abstract":"For problems requiring cooperation, many multiagent systems implement solutions among either individual agents or across an entire population towards a common goal. Multiagent teams are primarily studied when in conflict; however, organizational psychology (OP) highlights the benefits of teams among human populations for learning how to coordinate and cooperate. In this paper, we propose a new model of multiagent teams for reinforcement learning (RL) agents inspired by OP and early work on teams in artificial intelligence. We validate our model using complex social dilemmas that are popular in recent multiagent RL and find that agents divided into teams develop cooperative pro-social policies despite incentives to not cooperate. Furthermore, agents are better able to coordinate and learn emergent roles within their teams and achieve higher rewards compared to when the interests of all agents are aligned.","doi":"10.24963/ijcai.2022/65","url":"https://ijcai.org/proceedings/2022/65","pdf":"https://ijcai.org/proceedings/2022/0065.pdf","markdown":"https://ijcai.org/proceedings/2022/65.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/65","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":65,"pages":{"first":454,"last":460},"keywords":["Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence","Agent-based and Multi-agent Systems: Coordination and Cooperation","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987059","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985202","length_sec":711,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anastassacos et al., 2020] N. Anastassacos, S. Hailes, and M. Musolesi. Partner selection for the emergence of coop- eration in multi-agent systems using reinforcement learn- ing. In AAAI, 2020.","title":"Partner selection for the emergence of coop-eration in multi-agent systems using reinforcement learning","authors":"N Anastassacos, S Hailes, M Musolesi","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anastassacos et al., 2021] N. Anastassacos, J. Garcia, S. Hailes, and M. Musolesi. Cooperation and reputation dynamics with reinforcement learning. In AAMAS, 2021.","title":"Cooperation and reputation dynamics with reinforcement learning","authors":"N Anastassacos, J Garcia, S Hailes, M Musolesi","year":"2021","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andrejczuk et al., 2016] E. Andrejczuk, J. A. Rodriguez- Aguilar, and C. Sierra. A concise review on multiagent teams: contributions and research opportunities. Multi- Agent Systems and Agreement Technologies, 2016.","title":"A concise review on multiagent teams: contributions and research opportunities","authors":"E Andrejczuk, J A Rodriguez-Aguilar, C Sierra","year":"2016","venue":"Multi-Agent Systems and Agreement Technologies","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baker et al., 2019] B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch. Emergent tool use from multi-agent autocurricula. In ICLR, 2019.","title":"Emergent tool use from multi-agent autocurricula","authors":"B Baker, I Kanitscheider, T Markov, Y Wu, G Powell, B Mcgrew, I Mordatch","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baker, 2020] B. Baker. Emergent reciprocity and team formation from randomized uncertain social preferences. NeurIPS, 2020.","title":"Emergent reciprocity and team formation from randomized uncertain social preferences","authors":"B Baker","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dafoe et al., 2021] A. Dafoe, Y. Bachrach, G. Hadfield, E. Horvitz, K. Larson, and T. Graepel. Cooperative AI: machines must learn to find common ground. Nature, 2021.","title":"Cooperative AI: machines must learn to find common ground","authors":"A Dafoe, Y Bachrach, G Hadfield, E Horvitz, K Larson, T Graepel","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deka and Sycara, 2021] A. Deka and K. Sycara. Natural emergence of heterogeneous strategies in artificially intel- ligent competitive teams. In Int. Conf. on Swarm Intelli- gence, 2021.","title":"Natural emergence of heterogeneous strategies in artificially intelligent competitive teams","authors":"Sycara Deka, ] A Deka, K Sycara","year":"2021","venue":"Int. Conf. on Swarm Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grosz and Kraus, 1996] B. Grosz and S. Kraus. Collabora- tive plans for complex group action. Artif. Intell., 1996.","title":"Collaborative plans for complex group action","authors":"B Grosz, S Kraus","year":"1996","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grosz and Sidner, 1988] B. J. Grosz and C. L. Sidner. Plans for discourse. Technical report, BBN Labs, 1988.","title":"Plans for discourse","authors":"B J Grosz, C L Sidner","year":"1988","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hughes et al., 2018] E. Hughes, J. Z. Leibo, M. Phillips, K. Tuyls, E. A. Duéñez-Guzmán, A. Castañeda, I. Dun- ning, T. Zhu, K. R. McKee, R. Koster, H. Roff, and T. Graepel. Inequity aversion improves cooperation in in- tertemporal social dilemmas. In NeurIPS, 2018.","title":"Inequity aversion improves cooperation in intertemporal social dilemmas","authors":"E Hughes, J Z Leibo, M Phillips, K Tuyls, E A Duéñez-Guzmán, A Castañeda, I Dunning, T Zhu, K R Mckee, R Koster, H Roff, T Graepel","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaderberg et al., 2019] M. Jaderberg, W. Czarnecki, I. Dun- ning, L. Marris, G. Lever, A. Castañeda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, N. Sonnerat, T. Green, L. Deason, J. Z. Leibo, D. Silver, D. Hassabis, K. Kavukcuoglu, and T. Graepel. Human-level perfor- mance in 3d multiplayer games with population-based re- inforcement learning. Science, 364:859 -865, 2019.","title":"Human-level performance in 3d multiplayer games with population-based reinforcement learning","authors":"M Jaderberg, W Czarnecki, I Dunning, L Marris, G Lever, A Castañeda, C Beattie, N C Rabinowitz, A S Morcos, A Ruderman, N Sonnerat, T Green, L Deason, J Z Leibo, D Silver, D Hassabis, K Kavukcuoglu, T Graepel","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaques et al., 2019] N. Jaques, A. Lazaridou, E. Hughes, C ¸. Gülc ¸ehre, P. A. Ortega, D. Strouse, J. Z. Leibo, and N. D. Freitas. Social influence as intrinsic motivation for multi- agent deep reinforcement learning. In ICML, 2019.","title":"Social influence as intrinsic motivation for multiagent deep reinforcement learning","authors":"N Jaques, A Lazaridou, E Hughes, C ¸ Gülc ¸ehre, P A Ortega, D Strouse, J Z Leibo, N D Freitas","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kitano et al., 1997] H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, and E. Osawa. Robocup: The robot world cup initiative. In AGENTS '97, 1997.","title":"Robocup: The robot world cup initiative","authors":"H Kitano, M Asada, Y Kuniyoshi, I Noda, E Osawa","year":"1997","venue":"AGENTS '97","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kraemer and Banerjee, 2016] L. Kraemer and B. Banerjee. Multi-agent reinforcement learning as a rehearsal for de- centralized planning. Neurocomputing, 190:82-94, 2016.","title":"Multi-agent reinforcement learning as a rehearsal for decentralized planning","authors":"L Kraemer, B Banerjee","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Leibo et al., 2017] J. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multiagent reinforcement learning in sequential social dilemmas. AAMAS, 2017.","title":"Multiagent reinforcement learning in sequential social dilemmas","authors":"J Leibo, V Zambaldi, M Lanctot, J Marecki, T Graepel","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Macke et al., 2021] W. Macke, R. Mirsky, and P. Stone. Ex- pected value of communication for planning in ad hoc teamwork. In AAAI-21, 2021.","title":"Expected value of communication for planning in ad hoc teamwork","authors":"W Macke, R Mirsky, P Stone","year":"2021","venue":"AAAI-21","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McKee et al., 2020] K.R. McKee, I. Gemp, B. McWilliams, E.A. Duéñez-Guzmán, E. Hughes, and J. Z. Leibo. So- cial diversity and social preferences in mixed-motive rein- forcement learning. AAMAS, 2020.","title":"Social diversity and social preferences in mixed-motive reinforcement learning","authors":"K R Mckee, I Gemp, B Mcwilliams, E A Duéñez-Guzmán, E Hughes, J Z Leibo","year":"2020","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Ku- maran, D. Wierstra, S. Legg, and D. Hassabis. Human- level control through deep reinforcement learning. Nature, 518:529-533, 2015.","title":"Humanlevel control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A A Rusu, J Veness, M G Bellemare, A Graves, M A Riedmiller, A Fidjeland, G Ostrovski, S Petersen, C Beattie, A Sadik, I Antonoglou, H King, D Kumaran, D Wierstra, S Legg, D Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Muthukrishna, 2017] M. Muthukrishna. Corruption, coop- eration, and the evolution of prosocial institutions. Evo- nomics, 2017.","title":"Corruption, cooperation, and the evolution of prosocial institutions","authors":"M Muthukrishna","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nisioti and Moulin-Frier, 2020] E. Nisioti and C. Moulin- Frier. Grounding artificial intelligence in the origins of human behavior. ArXiv, abs/2012.08564, 2020.","title":"Grounding artificial intelligence in the origins of human behavior","authors":"E Nisioti, C Moulin-Frier","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pollack, 1990] M. E. Pollack. Plans as complex mental atti- tudes. In Intentions in Communication, 1990.","title":"Plans as complex mental attitudes","authors":"M E Pollack","year":"1990","venue":"Intentions in Communication","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Radke et al., 2022] D. Radke, K. Larson, and T. Brecht. The importance of credo in multiagent learning. Adaptive and Learning Agents Workshop at AAMAS, 2022.","title":"The importance of credo in multiagent learning","authors":"D Radke, K Larson, T Brecht","year":"2022","venue":"Adaptive and Learning Agents Workshop at AAMAS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rapoport, 1974] A. Rapoport. Prisoner's dilemma -recol- lections and observations. In Game Theory as a Theory of a Conflict Resolution, pages 17-34. Springer, 1974.","title":"Prisoner's dilemma -recollections and observations","authors":"A Rapoport","year":"1974","venue":"Game Theory as a Theory of a Conflict Resolution","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ryu et al., 2021] H. Ryu, H. Shin, and J. Park. Coopera- tive and competitive biases for multi-agent reinforcement learning. In AAMAS, 2021.","title":"Cooperative and competitive biases for multi-agent reinforcement learning","authors":"H Ryu, H Shin, J Park","year":"2021","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schnell et al., 2021] E. Schnell, R. Schimmelpfennig, and M. Muthukrishna. The size of the stag determines the level of cooperation. bioRxiv, 2021.","title":"The size of the stag determines the level of cooperation","authors":"E Schnell, R Schimmelpfennig, M Muthukrishna","year":"2021","venue":"bioRxiv","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tambe, 1997] M. Tambe. Towards flexible teamwork. J. Artif. Intell. Res., 7:83-124, 1997.","title":"Towards flexible teamwork","authors":"M Tambe","year":"1997","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vinitsky et al., 2019] E. Vinitsky, N. Jaques, J. Leibo, A. Castenada, and E. Hughes. An open source implementation of sequential social dilemma games. https://github.com/eugenevinitsky/ sequential social dilemma games/issues/182, 2019.","title":"An open source implementation of sequential social dilemma games","authors":"E Vinitsky, N Jaques, J Leibo, A Castenada, E Hughes","year":"2019","venue":null,"doi":null,"url":"https://github.com/eugenevinitsky/sequentialsocialdilemmagames/issues/182","internal_url":null},{"position":29,"raw":"[Wang et al., 2019] J. X. Wang, E. Hughes, C. Fernando, W. M. Czarnecki, E. A. Duéñez-Guzmán, and J. Z. Leibo. Evolving intrinsic motivations for altruistic behavior. In AAMAS-19, pages 683-692, 2019.","title":"Evolving intrinsic motivations for altruistic behavior","authors":"J X Wang, E Hughes, C Fernando, W M Czarnecki, E A Duéñez-Guzmán, J Z Leibo","year":"2019","venue":"AAMAS-19","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zaccaro et al., 2020] S. J. Zaccaro, S. Dubrow, E. M. Tor- res, and L. Campbell. Multiteam systems: An integrated review and comparison of different forms. Annual Review of Organizational Psychology and Organizational Behav- ior, 2020.","title":"Multiteam systems: An integrated review and comparison of different forms","authors":"S J Zaccaro, S Dubrow, E M Torres, L Campbell","year":"2020","venue":"Annual Review of Organizational Psychology and Organizational Behavior","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}