{"type":"ScholarlyArticle","title":"Imitation Learning from Video by Leveraging Proprioception","authors":[{"name":"Faraz Torabi","slug":"faraz-torabi","given":"Faraz","family":"Torabi","affiliation":"The University of Texas at Austin"},{"name":"Garrett Warnell","slug":"garrett-warnell","given":"Garrett","family":"Warnell","affiliation":"Army Research Laboratory"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"The University of Texas at Austin"}],"abstract":"Classically, imitation learning algorithms have been developed for idealized situations, e.g., the demonstrations are often required to be collected in the exact same environment and usually include the demonstrator's actions. Recently, however, the research community has begun to address some of these shortcomings by offering algorithmic solutions that enable imitation learning from observation (IfO), e.g., learning to perform a task from visual demonstrations that may be in a different environment and do not include actions. Motivated by the fact that agents often also have access to their own internal states (i.e., proprioception), we propose and study an IfO algorithm that leverages this information in the policy learning process. The proposed architecture learns policies over proprioceptive state representations and compares the resulting trajectories visually to the demonstration data. We experimentally test the proposed technique on several MuJoCo domains and show that it outperforms other imitation from observation algorithms by a large margin.","doi":"10.24963/ijcai.2019/497","url":"https://ijcai.org/proceedings/2019/497","pdf":"https://ijcai.org/proceedings/2019/0497.pdf","markdown":"https://ijcai.org/proceedings/2019/497.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/497","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":497,"pages":{"first":3585,"last":3591},"keywords":["Machine Learning: Reinforcement Learning","Robotics: Behavior and Control"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the twenty-first International Con- ference on Machine learning, page 1. ACM, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the twenty-first International Conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In ICML, pages 214-223, 2017.","title":"Wasserstein generative adversarial networks","authors":"Martin Arjovsky, Soumith Chintala, Léon Bottou","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aytar et al., 2018] Yusuf Aytar, Tobias Pfaff, David Bud- den, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In NeurIPS, pages 2935-2945, 2018.","title":"Playing hard exploration games by watching youtube","authors":"Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, Nando De Freitas","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bagnell et al., 2007] JA Bagnell, Joel Chestnutt, David M Bradley, and Nathan D Ratliff. Boosting structured pre- diction for imitation learning. In Advances in Neural In- formation Processing Systems, pages 1153-1160, 2007.","title":"Boosting structured prediction for imitation learning","authors":"Bagnell","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bain and Sammut, 1995] Michael Bain and Claude Sam- mut. A framework for behavioral cloning. Machine In- telligence 14, 1995.","title":"A framework for behavioral cloning","authors":"Sammut Bain, Michael Bain, Claude Sammut","year":"1995","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bain and Sommut, 1999] Michael Bain and Claude Som- mut. A framework for behavioural claning. Machine In- telligence 15, 15:103, 1999.","title":"A framework for behavioural claning","authors":"Michael Bain, Claude Sommut","year":"1999","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Baker et al., 2009] Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):329-349, 2009.","title":"Action understanding as inverse planning","authors":"Chris L Baker, Rebecca Saxe, Joshua B Tenenbaum","year":"2009","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, 2016. [","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Coumans and Bai, 2016 2017]","title":null,"authors":"Bai Coumans","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016-2017.","title":"Pybullet, a python module for physics simulation for games, robotics and machine learning","authors":"Erwin Coumans, Yunfei Bai","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Daftry et al., 2016] Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monocular reactive mav control. In International Sym- posium on Experimental Robotics, pages 3-11. Springer, 2016.","title":"Learning transferable policies for monocular reactive mav control","authors":"Daftry","year":"2016","venue":"International Symposium on Experimental Robotics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Edwards et al., 2019] Ashley D Edwards, Himanshu Sahni, Yannick Schroeker, and Charles L Isbell. Imitating latent policies from observation. ICML, 2019.","title":"Imitating latent policies from observation","authors":"Himanshu Ashley D Edwards, Yannick Sahni, Charles L Schroeker, Isbell","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fu et al., 2018] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In ICLR, 2018.","title":"Learning robust rewards with adverserial inverse reinforcement learning","authors":"Justin Fu, Katie Luo, Sergey Levine","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 17(1):2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Goo and Niekum, 2019] Wonjoon Goo and Scott Niekum. One-shot learning of multi-step tasks from observation via activity localization in auxiliary video. ICRA, 2019.","title":"Wonjoon Goo and Scott Niekum. One-shot learning of multi-step tasks from observation via activity localization in auxiliary video","authors":"Niekum Goo","year":"2019","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in Neural Informa- tion Processing Systems, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gupta et al., 2017] Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learn- ing. In International Conference on Learning Representa- tions, 2017.","title":"Learning invariant feature spaces to transfer skills with reinforcement learning","authors":"Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, Sergey Levine","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565- 4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kostrikov et al., 2019] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Address- ing sample inefficiency and reward bias in adversarial imitation learning. In ICLR, 2019.","title":"Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning","authors":"Ilya Kostrikov, Krishna Kumar, Debidatta Agrawal, Sergey Dwibedi, Jonathan Levine, Tompson","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2018] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. 2018.","title":"Imitation from observation: Learning to imitate behaviors from raw video via context translation","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Merel et al., 2017] Josh Merel, Yuval Tassa, Sriram Srini- vasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201, 2017.","title":"Learning human behaviors from motion capture by adversarial imitation","authors":"Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, Nicolas Heess","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Misra et al., 2016] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, pages 527- 544. Springer, 2016.","title":"Shuffle and learn: unsupervised learning using temporal order verification","authors":"Ishan Misra, Lawrence Zitnick, Martial Hebert","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Osa et al., 2018] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learn- ing. Foundations and Trends R in Robotics, 7(1-2):1-179, 2018.","title":"An algorithmic perspective on imitation learning","authors":"Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, Jan Peters","year":"2018","venue":"Foundations and Trends R in Robotics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pavse et al., 2019] Brahma Pavse, Faraz Torabi, Josiah Hanna, Garrett Warnell, and Peter Stone. Ridm: Rein- forced inverse dynamics modeling for learning from a sin- gle observed demonstration. In International Conference on Machine Learning Workshop on Imitation, Intent, and Interaction (I3), 2019.","title":"Ridm: Reinforced inverse dynamics modeling for learning from a single observed demonstration","authors":"Pavse","year":"2019","venue":"International Conference on Machine Learning Workshop on Imitation, Intent, and Interaction","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Qureshi et al., 2019] Ahmed H. Qureshi, Byron Boots, and Michael C. Yip. Adversarial imitation via variational in- verse reinforcement learning. In ICLR, 2019.","title":"Adversarial imitation via variational inverse reinforcement learning","authors":"Ahmed H Qureshi, Byron Boots, Michael C Yip","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey J Gordon, Drew Bagnell","year":"2011","venue":"Interna-tional Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sasaki et al., 2019] Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. In ICLR, 2019.","title":"Sample efficient imitation learning for continuous control","authors":"Fumihiro Sasaki, Tetsuya Yohira, Atsuo Kawaguchi","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Schaal, 1997] Stefan Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems, pages 1040-1046, 1997.","title":"Learning from demonstration","authors":"Stefan Schaal, Schaal","year":"1997","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sermanet et al., 2018] Pierre Sermanet, Corey Lynch, Jas- mine Hsu, and Sergey Levine. Time-contrastive net- works: Self-supervised learning from multi-view observa- tion. In International Conference in Robotics and Automa- tion (ICRA), 2018.","title":"Time-contrastive networks: Self-supervised learning from multi-view observation","authors":"Sermanet","year":"2018","venue":"International Conference in Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Stadie et al., 2017] Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. In ICLR, 2017.","title":"Third-person imitation learning","authors":"Pieter Bradly C Stadie, Ilya Abbeel, Sutskever","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In IROS, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Torabi et al., 2018] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In IJCAI, pages 4950-4957, 2018.","title":"Behavioral cloning from observation","authors":"Faraz Torabi, Garrett Warnell, Peter Stone","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/687"},{"position":37,"raw":"[Torabi et al., 2019a] Faraz Torabi, Garrett Warnell, and Pe- ter Stone. Adversarial imitation learning from state-only demonstrations. In AAMAS, 2019.","title":"Adversarial imitation learning from state-only demonstrations","authors":"Torabi","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Torabi et al., 2019b] Faraz Torabi, Garrett Warnell, and Pe- ter Stone. Generative adversarial imitation from observa- tion. In International Conference on Machine Learning Workshop on Imitation, Intent, and Interaction (I3), 2019.","title":"Generative adversarial imitation from observation","authors":"Torabi","year":"2019","venue":"International Conference on Machine Learning Workshop on Imitation, Intent, and Interaction","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Torabi et al., 2019c] Faraz Torabi, Garrett Warnell, and Pe- ter Stone. Imitation learning from video by leveraging pro- prioception. arXiv preprint arXiv:1905.09335, 2019.","title":"Imitation learning from video by leveraging proprioception","authors":"Torabi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Torabi et al., 2019d] Faraz Torabi, Garrett Warnell, and Pe- ter Stone. Recent advances in imitation learning from ob- servation. In IJCAI. AAAI Press, 2019.","title":"Recent advances in imitation learning from observation","authors":"Torabi","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/882"},{"position":41,"raw":"[Vered et al., 2016] Mor Vered, Gal A Kaminka, and Sivan Biham. Online goal recognition through mirroring: Hu- mans and agents. In The Fourth Annual Conference on Advances in Cognitive Systems, 2016.","title":"Online goal recognition through mirroring: Humans and agents","authors":"Mor Vered, Sivan Kaminka, Biham","year":"2016","venue":"The Fourth Annual Conference on Advances in Cognitive Systems","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Vered et al., 2018] Mor Vered, Ramon Fraga Pereira, Maurício C Magnaguagno, Gal A Kaminka, and Felipe Meneguzzi. Towards online goal recognition combining goal mirroring and landmarks. In AAMAS, pages 2112- 2114, 2018.","title":"Towards online goal recognition combining goal mirroring and landmarks","authors":"Mor Vered, Ramon Fraga Pereira, C Maurício, Magnaguagno, Felipe Kaminka, Meneguzzi","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}