{"type":"ScholarlyArticle","title":"Learning Translations: Emergent Communication Pretraining for Cooperative Language Acquisition","authors":[{"name":"Dylan Cope","slug":"dylan-cope","given":"Dylan","family":"Cope","affiliation":"King's College London"},{"name":"Peter McBurney","slug":"peter-mcburney","given":"Peter","family":"McBurney","affiliation":"King's College London"}],"abstract":"In Emergent Communication (EC) agents learn to communicate with one another, but the protocols that they develop are specialised to their training community. This observation led to research into Zero-Shot Coordination (ZSC) for learning communication strategies that are robust to agents not encountered during training. However, ZSC typically assumes that no prior data is available about the agents that will be encountered in the zero-shot setting. In many cases, this presents an unnecessarily hard problem and rules out communication via preestablished conventions. We propose a novel AI challenge called a Cooperative Language Acquisition Problem (CLAP) in which the ZSC assumptions are relaxed by allowing a 'joiner' agent to learn from a dataset of interactions between agents in a target community. We propose and compare two methods for solving CLAPs: Behaviour Cloning (BC), and Emergent Communication pretraining and Translation Learning (ECTL), in which an agent is trained in self-play with EC and then learns to translate between an emergent protocol and the target community's protocol.","doi":"10.24963/ijcai.2024/5","url":"https://ijcai.org/proceedings/2024/5","pdf":"https://ijcai.org/proceedings/2024/0005.pdf","markdown":"https://ijcai.org/proceedings/2024/5.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/5","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":5,"pages":{"first":40,"last":48},"keywords":["Agent-based and Multi-agent Systems: MAS: Coordination and cooperation","Agent-based and Multi-agent Systems: MAS: Agent communication","Agent-based and Multi-agent Systems: MAS: Agent-based simulation and emergence","Machine Learning: ML: Multiagent Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Barrett et al., 2014] Samuel Barrett, Noa Agmon, Noam Hazon, Sarit Kraus, and Peter Stone. Communicating with unknown teammates. In Proceedings of the 2014 in- ternational conference on Autonomous agents and multi- agent systems, AAMAS '14, pages 1433-1434, Richland, SC, May 2014. International Foundation for Autonomous Agents and Multiagent Systems.","title":"Communicating with unknown teammates","authors":"Barrett","year":"2014","venue":"Proceedings of the 2014 international conference on Autonomous agents and multiagent systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bullard et al., 2020] Kalesha Bullard, Franziska Meier, Douwe Kiela, Joelle Pineau, and Jakob Foerster. Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations, December 2020. arXiv:2010.15896 [cs].","title":"Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations","authors":"Bullard","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bullard et al., 2021] Kalesha Bullard, Douwe Kiela, Franziska Meier, Joelle Pineau, and Jakob Foerster. Quasi-Equivalence Discovery for Zero-Shot Emergent Communication, June 2021. arXiv:2103.08067 [cs].","title":"Quasi-Equivalence Discovery for Zero-Shot Emergent Communication","authors":"Bullard","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cope and Schoots, 2020] Dylan Cope and Nandi Schoots. Learning to Communicate with Strangers via Channel Randomisation Methods. In The Emergent Communica- tion Workshop at NeurIPS, Vancouver, Canada, 2020.","title":"Dylan Cope and Nandi Schoots. Learning to Communicate with Strangers via Channel Randomisation Methods","authors":"Schoots Cope","year":"2020","venue":"The Emergent Communication Workshop at NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Foerster et al., 2016] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learn- ing to Communicate with Deep Multi-Agent Reinforce- ment Learning. In D. D. Lee and M. Sugiyama and U. V. Luxburg and I. Guyon and R. Garnett, editor, Advances in Neural Information Processing Systems 29, pages 2137- 2145.","title":"Learning to Communicate with Deep Multi-Agent Reinforcement Learning","authors":"Foerster","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goldman and Zilberstein, 2003] Claudia V. Goldman and Shlomo Zilberstein. Optimizing information exchange in cooperative multi-agent systems. In Proceedings of the 2nd International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 03), AAMAS '03, pages 137-144, New York, NY, USA, July 2003. Association for Computing Machinery.","title":"Optimizing information exchange in cooperative multi-agent systems","authors":"Claudia V Goldman, Shlomo Zilberstein","year":"2003","venue":"Proceedings of the 2nd International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 03), AAMAS '03","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goldman and Zilberstein, 2004] Claudia V. Goldman and Shlomo Zilberstein. Decentralized control of cooper- ative systems: categorization and complexity analysis. Journal of Artificial Intelligence Research, 22(1):143-174, November 2004.","title":"Decentralized control of cooperative systems: categorization and complexity analysis","authors":"Claudia V Goldman, Shlomo Zilberstein","year":"2004","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goldman and Zilberstein, 2008] Claudia V. Goldman and Shlomo Zilberstein. Communication-Based Decomposi- tion Mechanisms for Decentralized MDPs. Journal of Artificial Intelligence Research, 32:169-202, May 2008. arXiv:1111.0065 [cs].","title":"Communication-Based Decomposition Mechanisms for Decentralized MDPs","authors":"Claudia V Goldman, Shlomo Zilberstein","year":"2008","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2020] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. \"Other-Play\" for Zero-Shot Coordination. In Proceedings of the 37th International Conference on Machine Learning, pages 4399-4410. PMLR, November 2020. ISSN: 2640-3498.","title":"Other-Play\" for Zero-Shot Coordination","authors":"Hengyuan Hu, Adam Lerer, Alex Peysakhovich, Jakob Foerster","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2021] Hengyuan Hu, Adam Lerer, Brandon Cui, David Wu, Luis Pineda, Noam Brown, and Jakob Foer- ster. Off-Belief Learning. In Proceedings of the 38th In- ternational Conference on Machine Learning, PMLR 139, March 2021. arXiv: 2103.04000.","title":"Off-Belief Learning","authors":"Hengyuan Hu, Adam Lerer, Brandon Cui, David Wu, Luis Pineda, Noam Brown, Jakob Foerster","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hussein et al., 2017] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation Learn- ing: A Survey of Learning Methods. ACM Computing Surveys, 50(2):21:1-21:35, April 2017.","title":"Imitation Learning: A Survey of Learning Methods","authors":"Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, Chrisina Jayne","year":"2017","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jang et al., 2017] Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In 5th International Conference on Learning Representations (ICLR 17), Palais des Congrès Neptune, Toulon, France, 2017.","title":"Categorical Reparameterization with Gumbel-Softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2017","venue":"5th International Conference on Learning Representations (ICLR 17)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar et al., 2022] Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. Should I Run Offline Rein- forcement Learning or Behavioral Cloning? In Proceed- ings of the International Conference on Learning Repre- sentations (ICLR 22), 2022.","title":"Should I Run Offline Reinforcement Learning or Behavioral Cloning?","authors":"Aviral Kumar, Joey Hong, Anikait Singh, Sergey Levine","year":"2022","venue":"Proceedings of the International Conference on Learning Representations (ICLR 22)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lazaridou and Baroni, 2020] Angeliki Lazaridou and Marco Baroni. Emergent Multi-Agent Communication in the Deep Learning Era, July 2020. arXiv:2006.02419","title":"Emergent Multi-Agent Communication in the Deep Learning Era","authors":"Angeliki Lazaridou, Marco Baroni","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2023] Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Cooperative Open-ended Learning Framework for Zero-Shot Coordi- nation. In Proceedings of the 40th International Confer- ence on Machine Learning, pages 20470-20484. PMLR, July 2023. ISSN: 2640-3498.","title":"Cooperative Open-ended Learning Framework for Zero-Shot Coordination","authors":"Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, Wei Pan","year":"2023","venue":"Proceedings of the 40th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lowe et al., 2019] Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the Pit- falls of Measuring Emergent Communication. In The 18th International Conference on Autonomous Agents and Mul- tiagent Systems (AAMAS), 2019.","title":"On the Pitfalls of Measuring Emergent Communication","authors":"Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, Yann Dauphin","year":"2019","venue":"The 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maddison et al., 2017] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Con- tinuous Relaxation of Discrete Random Variables. In 5th International Conference on Learning Representations (ICLR 17), Palais des Congrès Neptune, Toulon, France, 2017.","title":"The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables","authors":"Chris J Maddison, Andriy Mnih, Yee Whye Teh","year":"2017","venue":"5th International Conference on Learning Representations (ICLR 17)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Michie et al., 1990] D Michie, M Bain, and J Mayes- Michie. Cognitive models from subcognitive skills. Knowledge-based Systems for Industrial Control, pages 71-99, January 1990.","title":"Cognitive models from subcognitive skills","authors":"M Michie, J Bain, Mayes-Michie","year":"1990","venue":"Knowledge-based Systems for Industrial Control","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mirsky et al., 2020] Reuth Mirsky, William Macke, Andy Wang, Harel Yedidsion, and Peter Stone. A Penny for Your Thoughts: The Value of Communication in Ad Hoc Teamwork. In Proceedings of the Twenty-Ninth Interna- tional Joint Conference on Artificial Intelligence, pages 254-260, Yokohama, Japan, July 2020.","title":"A Penny for Your Thoughts: The Value of Communication in Ad Hoc Teamwork","authors":"Reuth Mirsky, William Macke, Andy Wang, Harel Yedidsion, Peter Stone","year":"2020","venue":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/36"},{"position":21,"raw":"[Oliehoek and Amato, 2016] Frans A. Oliehoek and Christo- pher Amato. A Concise Introduction to Decentralized POMDPs. Springer International Publishing, 2016.","title":"A Concise Introduction to Decentralized POMDPs","authors":"Frans A Oliehoek, Christopher Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ossenkopf, 2020] Marie Ossenkopf. CoMaze: A cooper- ative game for zero-shot coordination. In The Emergent Communication Workshop at NeurIPS, 2020.","title":"CoMaze: A cooperative game for zero-shot coordination","authors":"Marie Ossenkopf","year":"2020","venue":"The Emergent Communication Workshop at NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rajaraman et al., 2020] Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the Fundamental Limits of Imitation Learning. In Advances in Neural Information Processing Systems, volume 33, pages 2914-2924.","title":"Toward the Fundamental Limits of Imitation Learning","authors":"Rajaraman","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ross and Bagnell, 2010] Stephane Ross and Drew Bagnell. Efficient Reductions for Imitation Learning. In Proceed- ings of the Thirteenth International Conference on Arti- ficial Intelligence and Statistics, pages 661-668. JMLR Workshop and Conference Proceedings, March 2010. ISSN: 1938-7228.","title":"Efficient Reductions for Imitation Learning","authors":"Ross, Bagnell, Stephane Ross, Drew Bagnell","year":"2010","venue":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sammut, 2010] Caude Sammut. Behavioral Cloning. In Claude Sammut and Geoffrey I. Webb, editors, Encyclo- pedia of Machine Learning, pages 93-97. Springer US, Boston, MA, 2010.","title":"Caude Sammut. Behavioral Cloning","authors":"Sammut","year":"2010","venue":"Encyclopedia of Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sarratt and Jhala, 2015] Trevor Sarratt and Arnav Jhala. The Role of Models and Communication in the Ad Hoc Multiagent Team Decision Problem. In Proceedings of the Third Annual Conference on Advances in Cognitive Sys- tems. Cognitive Systems Foundation, 2015.","title":"The Role of Models and Communication in the Ad Hoc Multiagent Team Decision Problem","authors":"Jhala Sarratt, Trevor Sarratt, Arnav Jhala","year":"2015","venue":"Proceedings of the Third Annual Conference on Advances in Cognitive Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stone et al., 2010] Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination. In Proceedings of the Twenty-Fourth AAAI Conference on Ar- tificial Intelligence, 2010.","title":"Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination","authors":"Peter Stone, A Gal, Sarit Kaminka, Jeffrey S Kraus, Rosenschein","year":"2010","venue":"Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sukhbaatar et al., 2016] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communi- cation with backpropagation. In Proceedings of the 30th International Conference on Neural Information Pro- cessing Systems, pages 2252-2260. Neural Information Processing Systems, 2016.","title":"Learning multiagent communication with backpropagation","authors":"Sukhbaatar","year":"2016","venue":"Proceedings of the 30th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tesauro, 1994] Gerald Tesauro. TD-Gammon, a Self- Teaching Backgammon Program, Achieves Master-Level Play. Neural Computation, 6(2):215-219, 1994.","title":"TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play","authors":"Gerald Tesauro","year":"1994","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wagner et al., 2003] Kyle Wagner, James A. Reggia, Juan Uriagereka, and Gerald S. Wilkinson. Progress in the Simulation of Emergent Communication and Language. Adaptive Behavior, 11(1):37-69, March 2003. Publisher: SAGE Publications Ltd STM.","title":"Progress in the Simulation of Emergent Communication and Language","authors":"Kyle Wagner, James A Reggia, Juan Uriagereka, Gerald S Wilkinson","year":"2003","venue":"Adaptive Behavior","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Widrow and Smith, 1964] Bernard Widrow and Fred W Smith. Pattern recognising control systems. Computer and information sciences, 1964.","title":"Pattern recognising control systems","authors":"Smith ; Widrow, Bernard Widrow, Fred W Smith","year":"1964","venue":"Computer and information sciences","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2022] Chao Yu, Akash Velu, Eugene Vinitsky, Ji- axuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The Surprising Effectiveness of PPO in Cooperative Multi- Agent Games. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks., New Orleans, USA, 2022.","title":"The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games","authors":"Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, Yi Wu","year":"2022","venue":"36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}