{"type":"ScholarlyArticle","title":"A Hierarchical Approach to Population Training for Human-AI Collaboration","authors":[{"name":"Yi Loo","slug":"yi-loo","given":"Yi","family":"Loo","affiliation":"Singapore University of Technology and Design (SUTD)"},{"name":"Chen Gong","slug":"chen-gong","given":"Chen","family":"Gong","affiliation":"Singapore University of Technology and Design (SUTD)"},{"name":"Malika Meghjani","slug":"malika-meghjani","given":"Malika","family":"Meghjani","affiliation":"Singapore University of Technology and Design (SUTD)"}],"abstract":"A major challenge for deep reinforcement learning (DRL) agents is to collaborate with novel partners that were not encountered by them during the training phase. This is specifically worsened by an increased variance in action responses when the DRL agents collaborate with human partners due to the lack of consistency in human behaviors. Recent work have shown that training a single agent as the best response to a diverse population of training partners significantly increases an agent's robustness to novel partners. We further enhance the population-based training approach by introducing a Hierarchical Reinforcement Learning (HRL) based method for Human-AI Collaboration. Our agent is able to learn multiple best-response policies as its low-level policy while at the same time, it learns a high-level policy that acts as a manager which allows the agent to dynamically switch between the low-level best-response policies based on its current partner. We demonstrate that our method is able to dynamically adapt to novel partners of different play styles and skill levels in the 2-player collaborative Overcooked game environment. We also conducted a human study in the same environment to test the effectiveness of our method when partnering with real human subjects. Code is available at https://gitlab.com/marvl-hipt/hipt.","doi":"10.24963/ijcai.2023/336","url":"https://ijcai.org/proceedings/2023/336","pdf":"https://ijcai.org/proceedings/2023/0336.pdf","markdown":"https://ijcai.org/proceedings/2023/336.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/336","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":336,"pages":{"first":3011,"last":3019},"keywords":["Machine Learning: ML: Deep reinforcement learning","Agent-based and Multi-agent Systems: MAS: Human-agent interaction","Humans and AI: HAI: Human-AI collaboration"],"videos":[],"references":[{"position":1,"raw":"[Alami et al., 2006] Rachid Alami, Aurélie Clodic, Vincent Montreuil, Emrah Akin Sisbot, and Raja Chatila. Toward human-aware robot task planning. In AAAI spring sympo- sium: to boldly go where no human-robot team has gone before, pages 39-46, 2006.","title":"Toward human-aware robot task planning","authors":"Rachid Alami, Aurélie Clodic, Vincent Montreuil, Emrah Akin Sisbot, Raja Chatila","year":"2006","venue":"AAAI spring symposium: to boldly go where no human-robot team has gone before","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 31, 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bard et al., 2020] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new fron- tier for ai research. Artificial Intelligence, 280:103216, 2020.","title":"The hanabi challenge: A new frontier for ai research","authors":"Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, Francis Song, Emilio Parisotto, Subhodeep Vincent Dumoulin, Edward Moitra, Hughes","year":"2020","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berner et al., 2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforce- ment learning. arXiv preprint arXiv:1912.06680, 2019.","title":"Dota 2 with large scale deep reinforcement learning","authors":"Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carroll et al., 2019] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.","title":"On the utility of learning about humans for human-ai coordination","authors":"Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, Anca Dragan","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Flet-Berliac, 2019] Yannis Flet-Berliac. The promise of hierarchical reinforcement learning. The Gradient, 2019. https://thegradient.pub/ the-promise-of-hierarchical-reinforcement-learning/.","title":"The promise of hierarchical reinforcement learning","authors":"Yannis Flet-Berliac","year":"2019","venue":"The Gradient","doi":null,"url":"https://thegradient.pub/the-promise-of-hierarchical-reinforcement-learning/","internal_url":null},{"position":7,"raw":"[Ghost Town Games, 2016] Ghost Town Games. Over- cooked. https://store.steampowered.com/app/448510/ Overcooked/, 2016. Accessed: 2023-01-18.","title":"Ghost Town Games. Overcooked","authors":null,"year":"2016","venue":null,"doi":null,"url":"https://store.steampowered.com/app/448510/Overcooked/,2016","internal_url":null},{"position":8,"raw":"[Gronauer and Diepold, 2022] Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a sur- vey. Artificial Intelligence Review, 55(2):895-943, 2022.","title":"Multi-agent deep reinforcement learning: a survey","authors":"Diepold Gronauer, Sven Gronauer, Klaus Diepold","year":"2022","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2020] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. \"other-play\" for zero-shot coordination. In International Conference on Machine Learning, pages 4399-4410. PMLR, 2020.","title":"other-play\" for zero-shot coordination","authors":"Hengyuan Hu, Adam Lerer, Alex Peysakhovich, Jakob Foerster","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2021] Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. Off-belief learning. In International Conference on Machine Learn- ing, pages 4369-4379. PMLR, 2021.","title":"Off-belief learning","authors":"Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, Jakob Foerster","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaques et al., 2019] Natasha Jaques, Angeliki Lazari- dou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pages 3040-3049. PMLR, 2019.","title":"Social influence as intrinsic motivation for multi-agent deep reinforcement learning","authors":"Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, Joel Z Strouse, Nando De Leibo, Freitas","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kirk et al., 2021] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of gener- alisation in deep reinforcement learning. arXiv preprint arXiv:2111.09794, 2021.","title":"A survey of generalisation in deep reinforcement learning","authors":"Robert Kirk, Amy Zhang, Edward Grefenstette, Tim Rocktäschel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klien et al., 2004] G. Klien, D.D. Woods, J.M. Bradshaw, R.R. Hoffman, and P.J. Feltovich. Ten challenges for mak- ing automation a \"team player\" in joint human-agent activ- ity. IEEE Intelligent Systems, 19(6):91-95, 2004.","title":"Ten challenges for making automation a \"team player\" in joint human-agent activity","authors":"G Klien, D D Woods, J M Bradshaw, R R Hoffman, P J Feltovich","year":"2004","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Alexander Li, Carlos Florensa, Ignasi Clav- era, and Pieter Abbeel. Sub-policy adaptation for hierar- chical reinforcement learning. In International Conference on Learning Representations, 2020.","title":"Sub-policy adaptation for hierarchical reinforcement learning","authors":"Alexander Li, Carlos Florensa, Ignasi Clavera, Pieter Abbeel","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lucas and Allen, 2022] Keane Lucas and Ross E Allen. Any-play: An intrinsic augmentation for zero-shot coor- dination. In Proceedings of the 21st International Con- ference on Autonomous Agents and Multiagent Systems, pages 853-861, 2022.","title":"Any-play: An intrinsic augmentation for zero-shot coordination","authors":"Keane Lucas, Ross E Allen","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lupu et al., 2021] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, pages 7204-7213. PMLR, 2021.","title":"Trajectory diversity for zero-shot coordination","authors":"Andrei Lupu, Brandon Cui, Hengyuan Hu, Jakob Foerster","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pateria et al., 2021] Shubham Pateria, Budhitama Sub- agdja, Ah-hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: A comprehensive survey. ACM Com- puting Surveys (CSUR), 54(5):1-35, 2021.","title":"Hierarchical reinforcement learning: A comprehensive survey","authors":"Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, Chai Quek","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shih et al., 2020] Andy Shih, Arjun Sawhney, Jovana Kondic, Stefano Ermon, and Dorsa Sadigh. On the criti- cal role of conventions in adaptive human-ai collaboration. In International Conference on Learning Representations, 2020.","title":"On the critical role of conventions in adaptive human-ai collaboration","authors":"Andy Shih, Arjun Sawhney, Jovana Kondic, Stefano Ermon, Dorsa Sadigh","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Stolle and Precup, 2002] Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Interna- tional Symposium on abstraction, reformulation, and ap- proximation, pages 212-223. Springer, 2002.","title":"Learning options in reinforcement learning","authors":"Precup Stolle, Martin Stolle, Doina Precup","year":"2002","venue":"International Symposium on abstraction, reformulation, and approximation","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Strouse et al., 2021] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collab- orating with humans without human data. Advances in Neural Information Processing Systems, 34:14502-14515, 2021.","title":"Collaborating with humans without human data","authors":"Strouse","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 1999] Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Richard S Sutton, Doina Precup, Satinder Singh","year":"1999","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. Feudal net- works for hierarchical reinforcement learning. In Inter- national Conference on Machine Learning, pages 3540- 3549. PMLR, 2017.","title":"Feudal networks for hierarchical reinforcement learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Nicolas Schaul, Max Heess, David Jaderberg, Koray Silver, Kavukcuoglu","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2020] Tonghan Wang, Tarun Gupta, Anuj Ma- hajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2020.","title":"Rode: Learning roles to decompose multi-agent tasks","authors":"Wang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2020] Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent rein- forcement learning with skill discovery. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1566-1574, 2020.","title":"Hierarchical cooperative multi-agent reinforcement learning with skill discovery","authors":"Yang","year":"2020","venue":"Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2021] Rui Zhao, Jinming Song, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Max- imum entropy population based training for zero-shot human-ai coordination. arXiv preprint arXiv:2112.11701, 2021.","title":"Maximum entropy population based training for zero-shot human-ai coordination","authors":"Rui Zhao, Jinming Song, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, Yang Wei","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}