{"type":"ScholarlyArticle","title":"A Penny for Your Thoughts: The Value of Communication in Ad Hoc Teamwork","authors":[{"name":"Reuth Mirsky","slug":"reuth-mirsky","given":"Reuth","family":"Mirsky","affiliation":"Department of Computer Science , The University of Texas at Austin"},{"name":"William Macke","slug":"william-macke","given":"William","family":"Macke","affiliation":"Department of Computer Science , The University of Texas at Austin"},{"name":"Andy Wang","slug":"andy-wang","given":"Andy","family":"Wang","affiliation":"Department of Computer Science , The University of Texas at Austin"},{"name":"Harel Yedidsion","slug":"harel-yedidsion","given":"Harel","family":"Yedidsion","affiliation":"Department of Computer Science , The University of Texas at Austin"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"Department of Computer Science , The University of Texas at Austin; Sony AI"}],"abstract":"In ad hoc teamwork, multiple agents need to collaborate without having knowledge about their teammates or their plans a priori. A common assumption in this research area is that the agents cannot communicate. However, just as two random people may speak the same language, autonomous teammates may also happen to share a communication protocol. This paper considers how such a shared protocol can be leveraged, introducing a means to reason about Communication in Ad Hoc Teamwork (CAT). The goal of this work is enabling improved ad hoc teamwork by judiciously leveraging the ability of the team to communicate.\r\nWe situate our study within a novel CAT scenario, involving tasks with multiple steps, where teammates' plans are unveiled over time. In this context, the paper proposes methods to reason about the timing and value of communication and introduces an algorithm for an ad hoc agent to leverage these methods. Finally, we introduces a new multiagent domain, the tool fetching domain, and we study how varying this domain's properties affects the usefulness of communication. Empirical results show the benefits of explicit reasoning about communication content and timing in ad hoc teamwork.","doi":"10.24963/ijcai.2020/36","url":"https://ijcai.org/proceedings/2020/36","pdf":"https://ijcai.org/proceedings/2020/0036.pdf","markdown":"https://ijcai.org/proceedings/2020/36.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/36","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":36,"pages":{"first":254,"last":260},"keywords":["Agent-based and Multi-agent Systems: Agent Communication","Agent-based and Multi-agent Systems: Coordination and Cooperation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26703/file_26703.m3u8","length_sec":858,"provider":"hls"}],"references":[{"position":1,"raw":"[Agmon and Stone, 2012] Noa Agmon and Peter Stone. Leading ad hoc agents in joint action settings with mul- tiple teammates. In AAMAS, pages 341-348, 2012.","title":"Leading ad hoc agents in joint action settings with multiple teammates","authors":"Agmon, Stone, Noa Agmon, Peter Stone","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Albrecht and Stone, 2017] Stefano V Albrecht and Peter Stone. Reasoning about hypothetical agent behaviours and their parameters. In AAMAS, pages 547-555, 2017.","title":"Reasoning about hypothetical agent behaviours and their parameters","authors":"V Stefano, Peter Albrecht, Stone","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Albrecht and Stone, 2018] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial In- telligence, 258:66-95, 2018.","title":"Autonomous agents modelling other agents: A comprehensive survey and open problems","authors":"V Stefano, Peter Albrecht, Stone","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Amir et al., 2016] Ofra Amir, Ece Kamar, Andrey Kolobov, and Barbara Grosz. Interactive teaching strategies for agent training. IJCAI, 2016.","title":"Interactive teaching strategies for agent training","authors":"Ofra Amir, Ece Kamar, Andrey Kolobov, Barbara Grosz","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/119"},{"position":5,"raw":"[Barrett et al., 2014] Samuel Barrett, Noa Agmon, Noam Hazon, Sarit Kraus, and Peter Stone. Communicating with unknown teammates. In AAMAS, pages 1433-1434, 2014.","title":"Communicating with unknown teammates","authors":"Barrett","year":"2014","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bisson et al., 2011] Francis Bisson, Froduald Kabanza, Ab- der Rezak Benaskeur, and Hengameh Irandoust. Provok- ing opponents to facilitate the recognition of their inten- tions. In AAAI, 2011.","title":"Provoking opponents to facilitate the recognition of their intentions","authors":"Francis Bisson, Froduald Kabanza, Abder Rezak Benaskeur, Hengameh Irandoust","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cakmak and Thomaz, 2012] Maya Cakmak and Andrea L Thomaz. Designing robot learners that ask good questions. In ACM/IEEE international conference on Human-Robot Interaction, 2012.","title":"Designing robot learners that ask good questions","authors":"Thomaz Cakmak, Maya Cakmak, Andrea L Thomaz","year":"2012","venue":"ACM/IEEE international conference on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chakraborty et al., 2017] Mithun Chakraborty, Kai Yee Phoebe Chua, Sanmay Das, and Brendan Juba. Coordinated vs. decentralized exploration in multi-agent multi-armed bandits. In IJCAI, pages 164-170, 2017.","title":"Coordinated vs. decentralized exploration in multi-agent multi-armed bandits","authors":"Mithun Chakraborty, Kai Yee, Phoebe Chua, Sanmay Das, Brendan Juba","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devin et al., 2019] Coline Devin, Daniel Geng, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Plan arith- metic: Compositional plan vectors for multi-task control. arXiv preprint arXiv:1910.14033, 2019.","title":"Plan arithmetic: Compositional plan vectors for multi-task control","authors":"Coline Devin, Daniel Geng, Pieter Abbeel, Trevor Darrell, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Epshteyn et al., 2008] Arkady Epshteyn, Adam Vogel, and Gerald DeJong. Active reinforcement learning. In Pro- ceedings of ICML, pages 296-303. ACM, 2008.","title":"Active reinforcement learning","authors":"Arkady Epshteyn, Adam Vogel, Gerald Dejong","year":"2008","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fern et al., 2007] Alan Fern, Sriraam Natarajan, Kshitij Ju- dah, and Prasad Tadepalli. A decision-theoretic model of assistance. In IJCAI, pages 1879-1884, 2007.","title":"A decision-theoretic model of assistance","authors":"Alan Fern, Sriraam Natarajan, Kshitij Judah, Prasad Tadepalli","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/303"},{"position":12,"raw":"[Foerster et al., 2016] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learn- ing to communicate with deep multi-agent reinforcement learning. In NIPS, pages 2137-2145, 2016.","title":"Learning to communicate with deep multi-agent reinforcement learning","authors":"Foerster","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Griffith et al., 2013] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with rein- forcement learning. In NIPS, pages 2625-2633, 2013.","title":"Policy shaping: Integrating human feedback with reinforcement learning","authors":"Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, Andrea L Thomaz","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hernandez-Leal et al., 2019] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. AAMAS, pages 1-48, 2019.","title":"A survey and critique of multiagent deep reinforcement learning","authors":"Hernandez-Leal","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Howard, 1966] Ronald A Howard. Information value the- ory. IEEE Transactions on systems science and cybernet- ics, 1966.","title":"Information value theory","authors":"A Ronald, Howard","year":"1966","venue":"IEEE Transactions on systems science and cybernetics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kapourchali and Banerjee, 2019] Masoumeh Heidari Kapourchali and Bonny Banerjee. State estimation via communication for monitoring. IEEE Transactions on Emerging Topics in Computational Intelligence, 2019.","title":"State estimation via communication for monitoring","authors":"Banerjee ; Masoumeh Heidari Kapourchali, Bonny Kapourchali, Banerjee","year":"2019","venue":"IEEE Transactions on Emerging Topics in Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kautz and Allen, 1986] Henry A Kautz and James F Allen. Generalized plan recognition. In AAAI, 1986.","title":"Generalized plan recognition","authors":"A Henry, James F Kautz, Allen","year":"1986","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Keren et al., 2014] Sarah Keren, Avigdor Gal, and Erez Karpas. Goal recognition design. In ICAPS, 2014.","title":"Goal recognition design","authors":"Sarah Keren, Avigdor Gal, Erez Karpas","year":"2014","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kulkarni et al., 2019] Anagha Kulkarni, Siddharth Srivas- tava, and Subbarao Kambhampati. A unified framework for planning in adversarial and cooperative environments. In AAAI, 2019.","title":"A unified framework for planning in adversarial and cooperative environments","authors":"Anagha Kulkarni, Siddharth Srivastava, Subbarao Kambhampati","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In NeurIPS, pages 6379-6390, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mirsky et al., 2018] Reuth Mirsky, Roni Stern, Kobi Gal, and Meir Kalech. Sequential plan recognition: An iterative approach to disambiguating between hypotheses. Artificial Intelligence, 260:51-73, 2018.","title":"Sequential plan recognition: An iterative approach to disambiguating between hypotheses","authors":"Reuth Mirsky, Roni Stern, Kobi Gal, Meir Kalech","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mordatch and Abbeel, 2018] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAAI, 2018.","title":"Emergence of grounded compositional language in multi-agent populations","authors":"Abbeel Mordatch, Igor Mordatch, Pieter Abbeel","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ravula et al., 2019] Manish Ravula, Shani Alkoby, and Pe- ter Stone. Ad hoc teamwork with behavior switching agents. In IJCAI, 2019.","title":"Ad hoc teamwork with behavior switching agents","authors":"Manish Ravula, Shani Alkoby, Peter Stone","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/78"},{"position":24,"raw":"[Roth et al., 2006] Maayan Roth, Reid Simmons, and Manuela Veloso. What to communicate? execution- time decision in multi-agent pomdps. In Distributed Au- tonomous Robotic Systems 7, pages 177-186. Springer, 2006.","title":"What to communicate? executiontime decision in multi-agent pomdps","authors":"Maayan Roth, Reid Simmons, Manuela Veloso","year":"2006","venue":"Distributed Autonomous Robotic Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shvo and McIlraith, 2020] Maayan Shvo and Sheila A McIlraith. Active goal recognition. In AAAI, 2020.","title":"Active goal recognition","authors":"Mcilraith Shvo, Maayan Shvo, Sheila A Mcilraith","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Singh, 1998] Munindar P Singh. Agent communication lan- guages: Rethinking the principles. Computer, 31(12):40- 47, 1998.","title":"Agent communication languages: Rethinking the principles","authors":"P Munindar, Singh","year":"1998","venue":"Computer","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Stone et al., 2010] Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In AAAI, 2010.","title":"Ad hoc autonomous agent teams: Collaboration without pre-coordination","authors":"Peter Stone, A Gal, Sarit Kaminka, Jeffrey S Kraus, Rosenschein","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stone et al., 2013] Peter Stone, Gal A Kaminka, Sarit Kraus, Jeffrey S Rosenschein, and Noa Agmon. Teaching and leading an ad hoc teammate: Collaboration without pre-coordination. Artificial Intelligence, 203:35-65, 2013.","title":"Teaching and leading an ad hoc teammate: Collaboration without pre-coordination","authors":"Peter Stone, A Gal, Sarit Kaminka, Jeffrey S Kraus, Noa Rosenschein, Agmon","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Torrey and Taylor, 2013] Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforce- ment learning. In AAMAS, pages 1053-1060, 2013.","title":"Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforcement learning","authors":"Torrey, Taylor","year":"2013","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2020] Rose E Wang, Sarah A Wu, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. Too many cooks: Coordinating multi- agent collaboration through inverse planning. AAMAS, 2020.","title":"Too many cooks: Coordinating multiagent collaboration through inverse planning","authors":"Sarah A Rose E Wang, James A Wu, Joshua B Evans, David C Tenenbaum, Max Parkes, Kleiman-Weiner","year":"2020","venue":"AAMAS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}