{"type":"ScholarlyArticle","title":"Adversarial Imitation Learning from Incomplete Demonstrations","authors":[{"name":"Mingfei Sun","slug":"mingfei-sun","given":"Mingfei","family":"Sun","affiliation":"Department of Computer Science and Engineering , Hong Kong University of Science and Technology"},{"name":"Xiaojuan Ma","slug":"xiaojuan-ma","given":"Xiaojuan","family":"Ma","affiliation":"Department of Computer Science and Engineering , Hong Kong University of Science and Technology"}],"abstract":"Imitation learning targets deriving a mapping from states to actions, a.k.a. policy, from expert demonstrations. Existing methods for imitation learning typically require any actions in the demonstrations to be fully available, which is hard to ensure in real applications. Though algorithms for learning with unobservable actions have been proposed, they focus solely on state information and over- look the fact that the action sequence could still be partially available and provide useful information for policy deriving. In this paper, we propose a novel algorithm called Action-Guided Adversarial Imitation Learning (AGAIL) that learns a pol- icy from demonstrations with incomplete action sequences, i.e., incomplete demonstrations. The core idea of AGAIL is to separate demonstrations into state and action trajectories, and train a policy with state trajectories while using actions as auxiliary information to guide the training whenever applicable. Built upon the Generative Adversarial Imitation Learning, AGAIL has three components: a generator, a discriminator, and a guide. The generator learns a policy with rewards provided by the discriminator, which tries to distinguish state distributions between demonstrations and samples generated by the policy. The guide provides additional rewards to the generator when demonstrated actions for specific states are available. We com- pare AGAIL to other methods on benchmark tasks and show that AGAIL consistently delivers com- parable performance to the state-of-the-art methods even when the action sequence in demonstrations is only partially available.","doi":"10.24963/ijcai.2019/487","url":"https://ijcai.org/proceedings/2019/487","pdf":"https://ijcai.org/proceedings/2019/0487.pdf","markdown":"https://ijcai.org/proceedings/2019/487.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/487","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":487,"pages":{"first":3513,"last":3519},"keywords":["Robotics: Learning in Robotics","Machine Learning: Reinforcement Learning","Machine Learning: Adversarial Machine Learning","Robotics: Behavior and Control"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In ICML, page 1. ACM, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"ICML, page 1","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bain and Sommut, 1999] Michael Bain and Claude Som- mut. A framework for behavioural cloning. Machine In- telligence, 15(15):103, 1999.","title":"A framework for behavioural cloning","authors":"Michael Bain, Claude Sommut","year":"1999","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baram et al., 2017] Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-end differentiable adversarial imitation learning. In ICML, pages 390-399, 2017.","title":"End-to-end differentiable adversarial imitation learning","authors":"Nir Baram, Oron Anschel, Itai Caspi, Shie Mannor","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bojarski et al., 2016] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Pra- soon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self- driving cars. arXiv preprint arXiv:1604.07316, 2016.","title":"End to end learning for selfdriving cars","authors":"Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brys et al., 2015] Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Tay- lor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, 2015.","title":"Reinforcement learning from demonstration through shaping","authors":"Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, Ann Nowé","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/472"},{"position":8,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In NeurIPS, pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Daftry et al., 2016] Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monocular reactive mav control. In International Sympo- sium on Experimental Robotics. Springer, 2016.","title":"Learning transferable policies for monocular reactive mav control","authors":"Daftry","year":"2016","venue":"International Symposium on Experimental Robotics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Finn et al., 2016] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal con- trol via policy optimization. In ICML, pages 49-58, 2016.","title":"Guided cost learning: Deep inverse optimal control via policy optimization","authors":"Finn","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gao et al., 2018] Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell, et al. Reinforcement learn- ing from imperfect demonstrations. arXiv preprint arXiv:1802.05313, 2018.","title":"Reinforcement learning from imperfect demonstrations","authors":"Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NeurIPS, pages 4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kang et al., 2018] Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In ICML, pages 2474-2483, 2018.","title":"Policy optimization with demonstrations","authors":"Kang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Yunzhu Li, Jiaming Song, and Stefano Er- mon. Infogail: Interpretable imitation learning from visual demonstrations. In NeurIPS, pages 3812-3822, 2017.","title":"Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations","authors":"Li","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Littman et al., 1995] Michael L Littman, Thomas L Dean, and Leslie Pack Kaelbling. On the complexity of solving markov decision problems. In Proceedings of the Eleventh conference on Uncertainty in Artificial Intelligence, pages 394-402.","title":"On the complexity of solving markov decision problems","authors":"Thomas L Michael L Littman, Leslie Dean, Kaelbling Pack","year":null,"venue":"Proceedings of the Eleventh conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Merel et al., 2017] Josh Merel, Yuval Tassa, Sriram Srini- vasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nico- las Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv:1707.02201, 2017.","title":"Learning human behaviors from motion capture by adversarial imitation","authors":"Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, Nicolas Heess","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, vol- ume 99, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algo- rithms for inverse reinforcement learning. In ICML, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pomerleau, 1991] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral Computation, 3(1):88-97, 1991.","title":"Efficient training of artificial neural networks for autonomous navigation","authors":"A Dean, Pomerleau","year":"1991","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Puterman, 2014] Martin L Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ross and Bagnell, 2010] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial in- telligence and statistics, pages 661-668, 2010.","title":"Efficient reductions for imitation learning","authors":"Ross, Bagnell ;, Stéphane Ross, Drew Bagnell","year":"2010","venue":"Proceedings of the thirteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceed- ings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey Gordon, Drew Bagnell","year":"2011","venue":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, volume 37, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Philipp Michael I Jordan, Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stadie et al., 2017] Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. arXiv preprint arXiv:1703.01703, 2017.","title":"Third-person imitation learning","authors":"Pieter Bradly C Stadie, Ilya Abbeel, Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Syed et al., 2008] Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In ICML, pages 1032-1039. ACM, 2008.","title":"Apprenticeship learning using linear programming","authors":"Syed","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Torabi et al., 2018] Faraz Torabi, Garrett Warnell, and Pe- ter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018.","title":"Behavioral cloning from observation","authors":"Faraz Torabi, Garrett Warnell, Peter Stone","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/687"},{"position":28,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ziebart et al., 2010] Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In ICML, pages 1255-1262, 2010.","title":"Modeling interaction via the principle of maximum causal entropy","authors":"Brian D Ziebart, J Andrew Bagnell, Anind K Dey","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}