{"type":"ScholarlyArticle","title":"Diversifying Training Pool Predictability for Zero-shot Coordination: A Theory of Mind Approach","authors":[{"name":"Dung Nguyen","slug":"dung-nguyen","given":"Dung","family":"Nguyen","affiliation":"Applied Artificial Intelligence Institute (A I 2 ), Deakin University"},{"name":"Hung Le","slug":"hung-le","given":"Hung","family":"Le","affiliation":"Applied Artificial Intelligence Institute (A I 2 ), Deakin University"},{"name":"Kien Do","slug":"kien-do","given":"Kien","family":"Do","affiliation":"Applied Artificial Intelligence Institute (A I 2 ), Deakin University"},{"name":"Sunil Gupta","slug":"sunil-gupta","given":"Sunil","family":"Gupta","affiliation":"Applied Artificial Intelligence Institute (A I 2 ), Deakin University"},{"name":"Svetha Venkatesh","slug":"svetha-venkatesh","given":"Svetha","family":"Venkatesh","affiliation":"Applied Artificial Intelligence Institute (A I 2 ), Deakin University"},{"name":"Truyen Tran","slug":"truyen-tran","given":"Truyen","family":"Tran","affiliation":"Applied Artificial Intelligence Institute (A I 2 ), Deakin University"}],"abstract":"The challenge in constructing artificial social agents is to enable adaptation ability to novel agents, and is called zero-shot coordination (ZSC). A promising approach is to train the adaptive agents by interacting with a diverse pool of collaborators, assuming that the greater the diversity in other agents seen during training, the better the generalisation. In this paper, we explore an alternative procedure by considering the behavioural predictability of collaborators, i.e. whether their actions and intentions are predictable, and use it to select a diverse set of agents for the training pool. More specifically, we develop a pool of agents through self-play training during which agents' behaviour evolves and has diversity in levels of behavioural predictability (LoBP) through its evolution. We construct an observer to compute the level of behavioural predictability for each version of the collaborators. To do so, the observer is equipped with the theory of mind (ToM) capability to learn to infer the actions and intentions of others. We then use an episodic memory based on the LoBP metric to maintain agents with different levels of behavioural predictability in the pool of agents. Since behaviours that emerge at the later training phase are more complex and meaningful, the memory is updated with the latest versions of training agents. Our extensive experiments demonstrate that LoBP-based diversity training leads to better ZSC than other diversity training methods.","doi":"10.24963/ijcai.2024/19","url":"https://ijcai.org/proceedings/2024/19","pdf":"https://ijcai.org/proceedings/2024/0019.pdf","markdown":"https://ijcai.org/proceedings/2024/19.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/19","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":19,"pages":{"first":166,"last":174},"keywords":["Machine Learning: ML: Multiagent Reinforcement Learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Albrecht and Stone, 2018] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial In- telligence, 258:66-95, 2018.","title":"Autonomous agents modelling other agents: A comprehensive survey and open problems","authors":"V Stefano, Peter Albrecht, Stone","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bain and Sammut, 1995] Michael Bain and Claude Sam- mut. A framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.","title":"A framework for behavioural cloning","authors":"Sammut Bain, Michael Bain, Claude Sammut","year":"1995","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bard et al., 2020] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new fron- tier for ai research. Artificial Intelligence, 280:103216, 2020.","title":"The hanabi challenge: A new frontier for ai research","authors":"Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, Francis Song, Emilio Parisotto, Subhodeep Vincent Dumoulin, Edward Moitra, Hughes","year":"2020","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baron-Cohen et al., 1985] Simon Baron-Cohen, Alan M Leslie, and Uta Frith. Does the autistic child have a \"the- ory of mind\"? Cognition, 21(1):37-46, 1985.","title":"Does the autistic child have a \"theory of mind","authors":"Baron-Cohen","year":"1985","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boutilier, 1996] Craig Boutilier. Planning, learning and co- ordination in multiagent decision processes. In TARK, vol- ume 96, pages 195-210.","title":"Planning, learning and coordination in multiagent decision processes","authors":"Craig Boutilier","year":null,"venue":"TARK","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bowling and McCracken, 2005] Michael Bowling and Pe- ter McCracken. Coordination and adaptation in im- promptu teams. In AAAI, volume 5, pages 53-58, 2005.","title":"Michael Bowling and Peter McCracken. Coordination and adaptation in impromptu teams","authors":"Mccracken Bowling","year":"2005","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Carroll et al., 2019] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32, 2019.","title":"On the utility of learning about humans for human-ai coordination","authors":"Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, Anca Dragan","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dafoe et al., 2021] Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Grae- pel. Cooperative ai: machines must learn to find common ground. Nature, 593(7857):33-36, 2021.","title":"Cooperative ai: machines must learn to find common ground","authors":"Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, Thore Graepel","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Duan et al., 2017] Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural informa- tion processing systems, 30, 2017.","title":"One-shot imitation learning","authors":"Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Openai, Jonas Ho, Ilya Schneider, Pieter Sutskever, Wojciech Abbeel, Zaremba","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In Inter- national Conference on Learning Representations, 2018. [Jaderberg et al., 2017] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine, ; Jaderberg","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jakubik et al., 2022] Johannes Jakubik, Michael Vössing, Niklas Kühl, Jannis Walk, and Gerhard Satzger. Data-centric artificial intelligence. arXiv preprint arXiv:2212.11854, 2022.","title":"Data-centric artificial intelligence","authors":"Johannes Jakubik, Michael Vössing, Niklas Kühl, Jannis Walk, Gerhard Satzger","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2023] Minqi Jiang, Tim Rocktäschel, and Ed- ward Grefenstette. General intelligence requires re- thinking exploration. Royal Society Open Science, 10(6):230539, 2023.","title":"Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethinking exploration","authors":"Jiang","year":"2023","venue":"Royal Society Open Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Knott et al., 2021] Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, Anca Dragan, and Rohin Shah. Evaluating the robustness of collaborative agents. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1560- 1562, 2021.","title":"Evaluating the robustness of collaborative agents","authors":"Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, Anca Dragan, Rohin Shah","year":"2021","venue":"Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lou et al., 2023] Xingzhou Lou, Jiaxian Guo, Junge Zhang, Jun Wang, Kaiqi Huang, and Yali Du. PECAN: Leverag- ing policy ensemble for context-aware zero-shot human-ai coordination. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Sys- tems, pages 679-688, 2023.","title":"PECAN: Leveraging policy ensemble for context-aware zero-shot human-ai coordination","authors":"Lou","year":"2023","venue":"Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2013] Xin Lu, Erik Wetter, Nita Bharti, Andrew J Tatem, and Linus Bengtsson. Approaching the limit of predictability in human mobility. Scientific reports, 3(1):2923, 2013.","title":"Approaching the limit of predictability in human mobility","authors":"Xin Lu, Erik Wetter, Nita Bharti, Andrew J Tatem, Linus Bengtsson","year":"2013","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lupu et al., 2021] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero- shot coordination. In International conference on machine learning, pages 7204-7213. PMLR, 2021.","title":"Trajectory diversity for zeroshot coordination","authors":"Andrei Lupu, Brandon Cui, Hengyuan Hu, Jakob Foerster","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mazumder et al., 2022] Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sud- nya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Han- nah Rose Kirk, et al. Dataperf: Benchmarks for data- centric ai development. arXiv preprint arXiv:2207.10062, 2022.","title":"Dataperf: Benchmarks for datacentric ai development","authors":"Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mirsky et al., 2022] Reuth Mirsky, Ignacio Carlucho, Ar- rasy Rahman, Elliot Fosong, William Macke, Mohan Srid- haran, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork: Definitions, methods, and open problems. In European Conference on Multiagent Systems, 2022.","title":"A survey of ad hoc teamwork: Definitions, methods, and open problems","authors":"Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, Stefano V Albrecht","year":"2022","venue":"European Conference on Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210-19222, 2021.","title":"Agent modelling under partial observability for deep reinforcement learning","authors":"Georgios Papoudakis, Filippos Christianos, Stefano Albrecht","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paul et al., 2021] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596-20607, 2021.","title":"Deep learning on a data diet: Finding important examples early in training","authors":"Paul","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Premack and Woodruff, 1978] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515-526, 1978.","title":"Does the chimpanzee have a theory of mind?","authors":"Woodruff ; ] David Premack, Guy Premack, Woodruff","year":"1978","venue":"Behavioral and brain sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rabinowitz et al., 2018] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Interna- tional conference on machine learning, pages 4218-4227.","title":"Machine theory of mind","authors":"Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S M Ali Eslami, Matthew Botvinick","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Song et al., 2010] Chaoming Song, Zehui Qu, Nicholas Blumm, and Albert-László Barabási. Limits of pre- dictability in human mobility. Science, 327(5968):1018- 1021, 2010.","title":"Limits of predictability in human mobility","authors":"Song","year":"2010","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Stone et al., 2010] Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1504-1509, 2010.","title":"Ad hoc autonomous agent teams: Collaboration without pre-coordination","authors":"Peter Stone, Gal Kaminka, Sarit Kraus, Jeffrey Rosenschein","year":"2010","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Strouse et al., 2021] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collab- orating with humans without human data. Advances in Neural Information Processing Systems, 34:14502-14515, 2021.","title":"Collaborating with humans without human data","authors":"Strouse","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Szot et al., 2023] Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, and Akshara Rai. Adaptive co- ordination in social embodied rearrangement. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Pro- ceedings of the 40th International Conference on Machine Learning, volume 202, pages 33365-33380. PMLR, 23- 29 Jul 2023.","title":"Adaptive coordination in social embodied rearrangement","authors":"Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, Akshara Rai","year":"2023","venue":"Proceedings of the 40th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tesauro, 1994] Gerald Tesauro. Td-gammon, a self- teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-219, 1994.","title":"Td-gammon, a selfteaching backgammon program, achieves master-level play","authors":"Gerald Tesauro","year":"1994","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Treutlein et al., 2021] Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. A new formal- ism, method and open issues for zero-shot coordination. In International Conference on Machine Learning, pages 10413-10423. PMLR, 2021.","title":"A new formalism, method and open issues for zero-shot coordination","authors":"Treutlein","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Whang et al., 2023] Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collection and qual- ity challenges in deep learning: A data-centric ai perspec- tive. The VLDB Journal, 32(4):791-813, 2023.","title":"Data collection and quality challenges in deep learning: A data-centric ai perspective","authors":"Steven Euijong Whang, Yuji Roh, Hwanjun Song, Jae-Gil Lee","year":"2023","venue":"The VLDB Journal","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yarats et al., 2022] Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13425, 2022.","title":"Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning","authors":"Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, Lerrel Pinto","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yu et al., 2022a] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learn- ing zero-shot cooperation with humans, assuming humans are biased. In The Eleventh International Conference on Learning Representations, 2022.","title":"Learning zero-shot cooperation with humans, assuming humans are biased","authors":"Yu","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yu et al., 2022b] Yu Yu, Shahram Khadivi, and Jia Xu. Can data diversity enhance learning generalization? In Pro- ceedings of the 29th international conference on computa- tional linguistics, pages 4933-4945, 2022.","title":"Can data diversity enhance learning generalization?","authors":"Yu","year":"2022","venue":"Proceedings of the 29th international conference on computational linguistics","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zha et al., 2023a] Daochen Zha, Zaid Pervaiz Bhat, Kwei- Herng Lai, Fan Yang, and Xia Hu. Data-centric ai: Per- spectives and challenges. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 945-948. SIAM, 2023.","title":"Data-centric ai: Perspectives and challenges","authors":"Zha","year":"2023","venue":"Proceedings of the 2023 SIAM International Conference on Data Mining (SDM)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zha et al., 2023b] Daochen Zha, Zaid Pervaiz Bhat, Kwei- Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158, 2023.","title":"Data-centric artificial intelligence: A survey","authors":"Zha","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhao et al., 2023] Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145-6153, 2023.","title":"Maximum entropy population-based training for zero-shot human-ai coordination","authors":"Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, Wei Yang","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}