{"type":"ScholarlyArticle","title":"Behavioral Cloning from Observation","authors":[{"name":"Faraz Torabi","slug":"faraz-torabi","given":"Faraz","family":"Torabi","affiliation":"The University of Texas at Austin"},{"name":"Garrett Warnell","slug":"garrett-warnell","given":"Garrett","family":"Warnell","affiliation":"U.S. Army Research Laboratory"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"The University of Texas at Austin"}],"abstract":"Humans often learn how to perform tasks via imitation: they observe others perform a task, and then very quickly infer the appropriate actions to take based on their observations. While extending this paradigm to autonomous agents is a well-studied problem in general, there are two particular aspects that have largely been overlooked: (1) that the learning is done from observation only (i.e., without explicit action information), and (2) that the learning is typically done very quickly. In this work, we propose a two-phase, autonomous imitation learning technique called behavioral cloning from observation (BCO), that aims to provide improved performance with respect to both of these aspects. First, we allow the agent to acquire experience in a self-supervised fashion. This experience is used to develop a model which is then utilized to learn a particular task by observing an expert perform that task without the knowledge of the specific actions taken. We experimentally compare BCO to imitation learning methods, including the state-of-the-art, generative adversarial imitation learning (GAIL) technique, and we show comparable task performance in several different simulation domains while exhibiting increased learning speed after expert trajectories become available.","doi":"10.24963/ijcai.2018/687","url":"https://ijcai.org/proceedings/2018/687","pdf":"https://ijcai.org/proceedings/2018/0687.pdf","markdown":"https://ijcai.org/proceedings/2018/687.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/687","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Robotics","track":"Main track","number":687,"pages":{"first":4950,"last":4957},"keywords":["Machine Learning: Deep Learning","Robotics: Learning in Robotics","Robotics: Behavior and Control"],"videos":[],"references":[{"position":1,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bain and Sommut, 1999] Michael Bain and Claude Som- mut. A framework for behavioural claning. Machine In- telligence 15, 15:103, 1999.","title":"A framework for behavioural claning","authors":"Michael Bain, Claude Sommut","year":"1999","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bojarski et al., 2016] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Pra- soon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self- driving cars. arXiv preprint arXiv:1604.07316, 2016.","title":"End to end learning for selfdriving cars","authors":"Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chebotar et al., 2017] Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-based and model- free updates for trajectory-centric reinforcement learning. arXiv preprint arXiv:1703.03078, 2017.","title":"Combining model-based and modelfree updates for trajectory-centric reinforcement learning","authors":"Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Daftry et al., 2016] Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monocular reactive mav control. In International Sym- posium on Experimental Robotics, pages 3-11. Springer, 2016.","title":"Learning transferable policies for monocular reactive mav control","authors":"Daftry","year":"2016","venue":"International Symposium on Experimental Robotics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Finn et al., 2016] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal con- trol via policy optimization. In International Conference on Machine Learning, pages 49-58, 2016.","title":"Guided cost learning: Deep inverse optimal control via policy optimization","authors":"Finn","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Giusti et al., 2016] Alessandro Giusti, Jérôme Guzzi, Dan C Cires ¸an, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 1(2):661-667, 2016.","title":"A machine learning approach to visual perception of forest trails for mobile robots","authors":"Alessandro Giusti, Jérôme Guzzi, Dan C Cires ¸an, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro","year":"2016","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gupta et al., 2017] Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learn- ing. arXiv preprint arXiv:1703.02949, 2017.","title":"Learning invariant feature spaces to transfer skills with reinforcement learning","authors":"Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hanna and Stone, 2017] Josiah P Hanna and Peter Stone. Grounded action transformation for robot learning in sim- ulation. In AAAI, pages 3834-3840, 2017.","title":"Grounded action transformation for robot learning in simulation","authors":"Stone ; Hanna, P Josiah, Peter Hanna, Stone","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565- 4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ho et al., 2016] Jonathan Ho, Jayesh Gupta, and Stefano Ermon. Model-free imitation learning with policy opti- mization. In International Conference on Machine Learn- ing, pages 2760-2769, 2016.","title":"Model-free imitation learning with policy optimization","authors":"Jonathan Ho, Jayesh Gupta, Stefano Ermon","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Konidaris, 2006] George D Konidaris. A framework for transfer in reinforcement learning. In ICML-06 Workshop on Structural Knowledge Transfer for Machine Learning, 2006.","title":"A framework for transfer in reinforcement learning","authors":"D George, Konidaris","year":"2006","venue":"ICML-06 Workshop on Structural Knowledge Transfer for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2017] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. arXiv preprint arXiv:1707.03374, 2017.","title":"Imitation from observation: Learning to imitate behaviors from raw video via context translation","authors":"Liu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Niekum et al., 2015] Scott Niekum, Sarah Osentoski, George Konidaris, Sachin Chitta, Bhaskara Marthi, and Andrew G Barto. Learning grounded finite-state representations from unstructured demonstrations. The In- ternational Journal of Robotics Research, 34(2):131-157, 2015.","title":"Learning grounded finite-state representations from unstructured demonstrations","authors":"Scott Niekum, Sarah Osentoski, George Konidaris, Sachin Chitta, Bhaskara Marthi, Andrew G Barto","year":"2015","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ross and Bagnell, 2010] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial in- telligence and statistics, pages 661-668, 2010.","title":"Efficient reductions for imitation learning","authors":"Ross, Bagnell ;, Stéphane Ross, Drew Bagnell","year":"2010","venue":"Proceedings of the thirteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey J Gordon, Drew Bagnell","year":"2011","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schaal, 1997] Stefan Schaal. Learning from demonstration. In Advances in neural information processing systems, pages 1040-1046, 1997.","title":"Learning from demonstration","authors":"Stefan Schaal, Schaal","year":"1997","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd In- ternational Conference on Machine Learning (ICML-15), pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML-15)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Taylor et al., 2008] Matthew E Taylor, Nicholas K Jong, and Peter Stone. Transferring instances for model- based reinforcement learning. In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases, pages 488-505. Springer, 2008.","title":"Transferring instances for modelbased reinforcement learning","authors":"Nicholas K Matthew E Taylor, Peter Jong, Stone","year":"2008","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}