{"type":"ScholarlyArticle","title":"Human-Agent Cooperation in Games under Incomplete Information through Natural Language Communication","authors":[{"name":"Shenghui Chen","slug":"shenghui-chen","given":"Shenghui","family":"Chen","affiliation":"University of Texas at Austin ,"},{"name":"Daniel Fried","slug":"daniel-fried","given":"Daniel","family":"Fried","affiliation":"Carnegie Mellon University"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"University of Texas at Austin ,"}],"abstract":"Developing autonomous agents that can strategize and cooperate with humans under information asymmetry is challenging without effective communication in natural language. We introduce a shared-control game, where two players collectively control a token in alternating turns to achieve a common objective under incomplete information. We formulate a policy synthesis problem for an autonomous agent in this game with a human as the other player. To solve this problem, we propose a communication-based approach comprising a language module and a planning module. The language module translates natural language messages into and from a finite set of flags, a compact representation defined to capture player intents. The planning module leverages these flags to compute a policy using an asymmetric information-set Monte Carlo tree search with flag exchange algorithm we present. We evaluate the effectiveness of this approach in a testbed based on Gnomes at Night, a search-and-find maze board game. Results of human subject experiments show that communication narrows the information gap between players and enhances human-agent cooperation efficiency with fewer turns.","doi":"10.24963/ijcai.2024/867","url":"https://ijcai.org/proceedings/2024/867","pdf":"https://ijcai.org/proceedings/2024/0867.pdf","markdown":"https://ijcai.org/proceedings/2024/867.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/867","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Human-Centred AI","track":"Human-Centred AI","number":867,"pages":{"first":7833,"last":7841},"keywords":["Natural Language Processing: NLP: Dialogue and interactive systems","Humans and AI: HAI: Human-AI collaboration","Planning and Scheduling: PS: Planning with Incomplete Information","Uncertainty in AI: UAI: Sequential decision making"],"videos":[],"references":[{"position":1,"raw":"[Bard et al., 2020] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The Hanabi challenge: A new fron- tier for AI research. Artificial Intelligence, 280:103216, 2020.","title":"The Hanabi challenge: A new frontier for AI research","authors":"Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, Francis Song, Emilio Parisotto, Subhodeep Vincent Dumoulin, Edward Moitra, Hughes","year":"2020","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bernstein et al., 2002] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The com- plexity of decentralized control of Markov decision pro- cesses. Mathematics of Operations Research, 27(4):819- 840, 2002.","title":"The complexity of decentralized control of Markov decision processes","authors":"Robert Daniel S Bernstein, Neil Givan, Shlomo Immerman, Zilberstein","year":"2002","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Browne et al., 2012] Cameron B Browne, Edward Pow- ley, Daniel Whitehouse, Simon M Lucas, Peter I Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of Monte Carlo tree search methods. Transactions on Com- putational Intelligence and AI in Games, 4(1):1-43, 2012.","title":"A survey of Monte Carlo tree search methods","authors":"Edward Cameron B Browne, Daniel Powley, Whitehouse, M Simon, Peter I Lucas, Philipp Cowling, Stephen Rohlfshagen, Diego Tavener, Spyridon Perez, Simon Samothrakis, Colton","year":"2012","venue":"Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Campbell et al., 2002] Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep Blue. Artificial intelligence, 2002.","title":"Deep Blue","authors":"Murray Campbell, Joseph Hoane, Feng-Hsiung Hsu","year":"2002","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carroll et al., 2019] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-AI coordination. In Advances in Neural Informa- tion Processing Systems, 2019.","title":"On the utility of learning about humans for human-AI coordination","authors":"Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, Anca Dragan","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cowling et al., 2012] Peter I Cowling, Edward J Powley, and Daniel Whitehouse. Information set Monte Carlo tree search. Transactions on Computational Intelligence and AI in Games, 4(2):120-143, 2012.","title":"Information set Monte Carlo tree search","authors":"Edward J Peter I Cowling, Daniel Powley, Whitehouse","year":"2012","venue":"Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cowling et al., 2015] Peter I Cowling, Daniel Whitehouse, and Edward J Powley. Emergent bluffing and inference with Monte Carlo tree search. In Conference on Computa- tional Intelligence and Games, pages 114-121, 2015.","title":"Emergent bluffing and inference with Monte Carlo tree search","authors":"Daniel Peter I Cowling, Edward J Whitehouse, Powley","year":"2015","venue":"Conference on Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goil et al., 2013] Aditya Goil, Matthew Derry, and Brenna D Argall. Using machine learning to blend human and robot controls for assisted wheelchair navigation. In International Conference on Rehabilitation Robotics, pages 1-6, 2013.","title":"Using machine learning to blend human and robot controls for assisted wheelchair navigation","authors":"Aditya Goil, Matthew Derry, Brenna D Argall","year":"2013","venue":"International Conference on Rehabilitation Robotics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodman, 2019] James Goodman. Re-determinizing infor- mation set Monte Carlo tree search in Hanabi. ArXiv preprint arXiv:1902.06075, 2019.","title":"Re-determinizing information set Monte Carlo tree search in Hanabi","authors":"James Goodman","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2017] He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. Learning symmetric collaborative dia- logue agents with dynamic knowledge graph embeddings. In Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017.","title":"Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings","authors":"He","year":"2017","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2018] He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. In Conference on Empirical Meth- ods in Natural Language Processing, 2018.","title":"Decoupling strategy and generation in negotiation dialogues","authors":"He","year":"2018","venue":"Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jacob et al., 2022] Athul Paul Jacob, David J Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and human- like gameplay with KL-regularized search. In Interna- tional Conference on Machine Learning, pages 9695- 9728, 2022.","title":"Modeling strong and humanlike gameplay with KL-regularized search","authors":"Athul Paul, Jacob, David J Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, Noam Brown","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In Eu- ropean Conference on Machine Learning, pages 282-293, 2006.","title":"Bandit based Monte-Carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lewis et al., 2017] Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? End-to-end learning of negotiation dialogues. In Conference on Empirical Methods in Natural Language Processing, 2017.","title":"Deal or no deal? End-to-end learning of negotiation dialogues","authors":"Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, Dhruv Batra","year":"2017","venue":"Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Palan and Schitter, 2018] Stefan Palan and Christian Schit- ter. Prolific. ac-A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17:22- 27, 2018.","title":"Stefan Palan and Christian Schitter. Prolific. ac-A subject pool for online experiments","authors":"Schitter Palan","year":"2018","venue":"Journal of Behavioral and Experimental Finance","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paquette et al., 2019] Philip Paquette, Yuchen Lu, Se- ton Steven Bocco, Max Smith, Satya O-G, Jonathan K Kummerfeld, Joelle Pineau, Satinder Singh, and Aaron C Courville. No-press diplomacy: Modeling multi-agent gameplay. In Advances in Neural Information Processing Systems, 2019.","title":"No-press diplomacy: Modeling multi-agent gameplay","authors":"Philip Paquette, Yuchen Lu, Seton Steven Bocco, Max Smith, O-G Satya, Jonathan K Kummerfeld, Joelle Pineau, Satinder Singh, Aaron C Courville","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peaceable Kingdom, 2016] Peaceable Kingdom. Gnomes at night. https://www.walmart.com/ip/Peaceable- Kingdom-Gnomes-at-Night-Game-2-to-4-Players-Ages- 6/585042377, 2016. Accessed: 2024-01-15.","title":"Peaceable Kingdom. Gnomes at night","authors":"Peaceable Kingdom","year":"2016","venue":null,"doi":null,"url":"https://www.walmart.com/ip/Peaceable-Kingdom-Gnomes-at-Night-Game-2-to-4-Players-Ages-6/585042377","internal_url":null},{"position":19,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354-359, 2017.","title":"Mastering the game of Go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szita et al., 2010] István Szita, Guillaume Chaslot, and Pieter Spronck. Monte-Carlo tree search in Settlers of Catan. In Advances in Computer Games, pages 21-32, 2010.","title":"Monte-Carlo tree search in Settlers of Catan","authors":"Szita","year":"2010","venue":"Advances in Computer Games","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}