{"type":"ScholarlyArticle","title":"Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrations","authors":[{"name":"Sangwon Seo","slug":"sangwon-seo","given":"Sangwon","family":"Seo","affiliation":"Rice University"},{"name":"Vaibhav V. Unhelkar","slug":"vaibhav-v-unhelkar","given":"Vaibhav V.","family":"Unhelkar","affiliation":"Rice University"}],"abstract":"We present Bayesian Team Imitation Learner (BTIL), an imitation learning algorithm to model the behavior of teams performing sequential tasks in Markovian domains. In contrast to existing multi-agent imitation learning techniques, BTIL explicitly models and infers the time-varying mental states of team members, thereby enabling learning of decentralized team policies from demonstrations of suboptimal teamwork. Further, to allow for sample- and label-efficient policy learning from small datasets, BTIL employs a Bayesian perspective and is capable of learning from semi-supervised demonstrations. We demonstrate and benchmark the performance of BTIL on synthetic multi-agent tasks as well as a novel dataset of human-agent teamwork. Our experiments show that BTIL can successfully learn team policies from demonstrations despite the influence of team members' (time-varying and potentially misaligned) mental states on their behavior.","doi":"10.24963/ijcai.2022/346","url":"https://ijcai.org/proceedings/2022/346","pdf":"https://ijcai.org/proceedings/2022/0346.pdf","markdown":"https://ijcai.org/proceedings/2022/346.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/346","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Humans and AI","track":"Main Track","number":346,"pages":{"first":2492,"last":2500},"keywords":["Machine Learning: Bayesian Learning","Robotics: Human Robot Interaction","Humans and AI: Human-AI Collaboration","Agent-based and Multi-agent Systems: Human-Agent Interaction","Uncertainty in AI: Sequential Decision Making"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986806","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984949","length_sec":360,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469- 483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beal, 2003] Matthew James Beal. Variational algorithms for ap- proximate Bayesian inference. PhD thesis, University College London, 2003.","title":"Variational algorithms for approximate Bayesian inference","authors":"Matthew James, Beal","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bhattacharyya et al., 2019] Raunak P Bhattacharyya, Derek J Phillips, Changliu Liu, Jayesh K Gupta, Katherine Driggs- Campbell, and Mykel J Kochenderfer. Simulating emergent prop- erties of human driving behavior using multi-agent reward aug- mented imitation learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 789-795. IEEE, 2019.","title":"Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning","authors":"Derek J Raunak P Bhattacharyya, Changliu Phillips, Liu, K Jayesh, Katherine Gupta, Driggs-Campbell, Mykel, Kochenderfer","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bogert and Doshi, 2018] Kenneth Bogert and Prashant Doshi. Multi-robot inverse reinforcement learning under occlusion with estimation of state transitions. Artificial Intelligence, 263:46-73, 2018.","title":"Multi-robot inverse reinforcement learning under occlusion with estimation of state transitions","authors":"Kenneth Bogert, Prashant Doshi","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2019] Daniel Brown, Wonjoon Goo, Prabhat Na- garajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observa- tions. In International Conference on Machine Learning, pages 783-792. PMLR, 2019.","title":"Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations","authors":"Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, Scott Niekum","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Letian Chen, Rohan Paleja, and Matthew Gom- bolay. Learning from suboptimal demonstration via self- supervised reward regression. In Conference on Robot Learning, pages 1262-1277. PMLR, 2021.","title":"Learning from suboptimal demonstration via selfsupervised reward regression","authors":"Chen","year":"2021","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choi and Kim, 2011] Jaedeug Choi and Kee-Eung Kim. Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research, 12:691-730, 2011.","title":"Inverse reinforcement learning in partially observable environments","authors":"Kim ; ] Jaedeug Choi, Kee-Eung Choi, Kim","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Choi et al., 2019] Sungjoon Choi, Kyungjae Lee, and Songhwai Oh. Robust learning from demonstrations with mixed quali- ties using leveraged Gaussian processes. IEEE Transactions on Robotics, 35(3):564-576, 2019.","title":"Robust learning from demonstrations with mixed qualities using leveraged Gaussian processes","authors":"Choi","year":"2019","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gangwani et al., 2020] Tanmay Gangwani, Joel Lehman, Qiang Liu, and Jian Peng. Learning belief representations for imitation learning in POMDPs. In Uncertainty in Artificial Intelligence, pages 1061-1071. PMLR, 2020.","title":"Learning belief representations for imitation learning in POMDPs","authors":"Tanmay Gangwani, Joel Lehman, Qiang Liu, Jian Peng","year":"2020","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Genera- tive adversarial imitation learning. Advances in Neural Informa- tion Processing Systems, 29:4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ivanovic et al., 2018] Boris Ivanovic, Edward Schmerling, Karen Leung, and Marco Pavone. Generative modeling of multimodal multi-human behavior. In 2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 3088- 3095. IEEE, 2018.","title":"Generative modeling of multimodal multi-human behavior","authors":"Boris Ivanovic, Edward Schmerling, Karen Leung, Marco Pavone","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Johnson and Willsky, 2014] Matthew Johnson and Alan Willsky. Stochastic variational inference for Bayesian time series models. In International Conference on Machine Learning, pages 1854- 1862. PMLR, 2014.","title":"Stochastic variational inference for Bayesian time series models","authors":"Matthew Johnson, Alan Willsky","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Le et al., 2017] Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Inter- national Conference on Machine Learning, pages 1995-2003. PMLR, 2017.","title":"Coordinated multi-agent imitation learning","authors":"Yisong Hoang M Le, Peter Yue, Patrick Carr, Lucey","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2019] Xiaomin Lin, Stephen C Adams, and Peter A Beling. Multi-agent inverse reinforcement learning for certain general-sum stochastic games. Journal of Artificial Intelligence Research, 66:473-502, 2019.","title":"Multi-agent inverse reinforcement learning for certain general-sum stochastic games","authors":"Xiaomin Lin, Stephen C Adams, Peter A Beling","year":"2019","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oliehoek and Amato, 2016] Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs, vol- ume 1. Springer, 2016.","title":"A concise introduction to decentralized POMDPs","authors":"Amato ; Oliehoek, A Frans, Christopher Oliehoek, Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Osa et al., 2018] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorith- mic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2):1-179, 2018.","title":"An algorithmic perspective on imitation learning","authors":"Osa","year":"2018","venue":"Foundations and Trends in Robotics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pomerleau, 1991] Dean A Pomerleau. Efficient training of artifi- cial neural networks for autonomous navigation. Neural compu- tation, 3(1):88-97, 1991.","title":"Efficient training of artificial neural networks for autonomous navigation","authors":"A Dean, Pomerleau","year":"1991","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Puterman, 1990] Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.","title":"Markov decision processes","authors":"Martin L Puterman","year":"1990","venue":"Handbooks in operations research and management science","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Salas et al., 2018] Eduardo Salas, Denise L Reyes, and Susan H McDaniel. The science of teamwork: Progress, reflections, and the road ahead. American Psychologist, 73(4):593, 2018.","title":"The science of teamwork: Progress, reflections, and the road ahead","authors":"Eduardo Salas, Denise L Reyes, Susan H Mcdaniel","year":"2018","venue":"American Psychologist","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seo et al., 2021] Sangwon Seo, Lauren R. Kennedy-Metz, Marco A. Zenati, Julie A. Shah, Roger D. Dias, and Vaibhav V. Unhelkar. Towards an AI coach to infer team mental model alignment in healthcare. In 2021 IEEE Conference on Cog- nitive and Computational Aspects of Situation Management (CogSIMA), pages 39-44, 2021.","title":"Towards an AI coach to infer team mental model alignment in healthcare","authors":"Sangwon Seo, Lauren R Kennedy-Metz, Marco A Zenati, Julie A Shah, Roger D Dias, Vaibhav V Unhelkar","year":"2021","venue":"IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2018] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 7472-7483, 2018.","title":"Multi-agent generative adversarial imitation learning","authors":"Jiaming Song, Hongyu Ren, Dorsa Sadigh, Stefano Ermon","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun and Ma, 2019] Mingfei Sun and Xiaojuan Ma. Adversarial imitation learning from incomplete demonstrations. In IJCAI International Joint Conference on Artificial Intelligence, page 3513, 2019.","title":"Adversarial imitation learning from incomplete demonstrations","authors":"Ma ; Sun, Mingfei Sun, Xiaojuan Ma","year":"2019","venue":"IJCAI International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/487"},{"position":23,"raw":"[Tannenbaum and Salas, 2020] Scott Tannenbaum and Eduardo Salas. Teams that work: the seven drivers of team effectiveness. Oxford University Press, 2020.","title":"Teams that work: the seven drivers of team effectiveness","authors":"Salas Tannenbaum, Scott Tannenbaum, Eduardo Salas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thomaz et al., 2016] Andrea Thomaz, Guy Hoffman, and Maya Cakmak. Computational human-robot interaction. Foundations and Trends in Robotics, 4(2-3):105-223, 2016.","title":"Computational human-robot interaction","authors":"Andrea Thomaz, Guy Hoffman, Maya Cakmak","year":"2016","venue":"Foundations and Trends in Robotics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Torabi et al., 2018] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4950-4957, 2018.","title":"Behavioral cloning from observation","authors":"Faraz Torabi, Garrett Warnell, Peter Stone","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/687"},{"position":26,"raw":"[Unhelkar and Shah, 2019] Vaibhav V Unhelkar and Julie A Shah. Learning models of sequential decision-making with partial spec- ification of agent behavior. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 2522-2530, 2019.","title":"Learning models of sequential decision-making with partial specification of agent behavior","authors":"V Vaibhav, Julie A Unhelkar, Shah","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021] Chen Wang, Claudia Pérez-D'Arpino, Danfei Xu, Li Fei-Fei, Karen Liu, and Silvio Savarese. Co-GAIL: Learn- ing diverse strategies for human-robot collaboration. In 5th An- nual Conference on Robot Learning, 2021.","title":"Co-GAIL: Learning diverse strategies for human-robot collaboration","authors":"Chen Wang, Claudia Pérez-D'arpino, Danfei Xu, Li Fei-Fei, Karen Liu, Silvio Savarese","year":"2021","venue":"5th Annual Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2021] Mengjiao Yang, Sergey Levine, and Ofir Nachum. TRAIL: Near-optimal imitation learning with subop- timal data. arXiv preprint arXiv:2110.14770, 2021.","title":"TRAIL: Near-optimal imitation learning with suboptimal data","authors":"Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. Advances in Neural Information Processing Systems, 34, 2021.","title":"Confidence-aware imitation learning from demonstrations with varying optimality","authors":"Zhang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zheng et al., 2014] Jiangchuan Zheng, Siyuan Liu, and Lionel M Ni. Robust Bayesian inverse reinforcement learning with sparse behavior noise. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 28, 2014.","title":"Robust Bayesian inverse reinforcement learning with sparse behavior noise","authors":"Zheng","year":"2014","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}