{"type":"ScholarlyArticle","title":"Leveraging Human Guidance for Deep Reinforcement Learning Tasks","authors":[{"name":"Ruohan Zhang","slug":"ruohan-zhang","given":"Ruohan","family":"Zhang","affiliation":"Department of Computer Science , University of Texas at Austin , USA"},{"name":"Faraz Torabi","slug":"faraz-torabi","given":"Faraz","family":"Torabi","affiliation":"Department of Computer Science , University of Texas at Austin , USA"},{"name":"Lin Guan","slug":"lin-guan","given":"Lin","family":"Guan","affiliation":"Department of Computer Science , University of Texas at Austin , USA"},{"name":"Dana H. Ballard","slug":"dana-h-ballard","given":"Dana H.","family":"Ballard","affiliation":"Department of Computer Science , University of Texas at Austin , USA"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"Department of Computer Science , University of Texas at Austin , USA"}],"abstract":"Reinforcement learning agents can learn to solve sequential decision tasks by interacting with the environment. Human knowledge of how to solve these tasks can be incorporated using imitation learning, where the agent learns to imitate human demonstrated decisions. However, human guidance is not limited to the demonstrations. Other types of guidance could be more suitable for certain tasks and require less human effort. This survey provides a high-level overview of five recent learning frameworks that primarily rely on human guidance other than conventional, step-by-step action demonstrations. We review the motivation, assumption, and implementation of each framework. We then discuss possible future research directions.","doi":"10.24963/ijcai.2019/884","url":"https://ijcai.org/proceedings/2019/884","pdf":"https://ijcai.org/proceedings/2019/0884.pdf","markdown":"https://ijcai.org/proceedings/2019/884.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/884","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Survey track","track":"Survey track","number":884,"pages":{"first":6339,"last":6346},"keywords":["Humans and AI: Human-AI Collaboration","Agent-based and Multi-agent Systems: Human-Agent Interaction","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Appren- ticeship learning via inverse reinforcement learning. In Pro- ceedings of the twenty-first international conference on Machine learning, page 1. ACM, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Ng Abbeel, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the twenty-first international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abel et al., 2017] David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in- the-loop reinforcement learning. NeurIPS Workshop on the Future of Interactive Learning Machines, 2017.","title":"Agent-agnostic human-inthe-loop reinforcement learning","authors":"David Abel, John Salvatier, Andreas Stuhlmüller, Owain Evans","year":"2017","venue":"NeurIPS Workshop on the Future of Interactive Learning Machines","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Akrour et al., 2014] Riad Akrour, Marc Schoenauer, Michèle Se- bag, and Jean-Christophe Souplet. Programming by feedback. In International Conference on Machine Learning, volume 32, pages 1503-1511. JMLR. org, 2014.","title":"Programming by feedback","authors":"Riad Akrour, Marc Schoenauer, Michèle Sebag, Jean-Christophe Souplet","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Amir et al., 2016] Ofra Amir, Ece Kamar, Andrey Kolobov, and Barbara J Grosz. Interactive teaching strategies for agent train- ing. In Proceedings of the Twenty-Fifth International Joint Con- ference on Artificial Intelligence, pages 804-811. AAAI Press, 2016.","title":"Interactive teaching strategies for agent training","authors":"Ofra Amir, Ece Kamar, Andrey Kolobov, Barbara J Grosz","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/119"},{"position":5,"raw":"[Andreas et al., 2017] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 166-175. JMLR. org, 2017.","title":"Modular multitask reinforcement learning with policy sketches","authors":"Jacob Andreas, Dan Klein, Sergey Levine","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Arakawa et al., 2018] Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, and Shin-ichi Maeda. Dqn-tamer: Human- in-the-loop reinforcement learning with intractable feedback. arXiv preprint arXiv:1810.11748, 2018.","title":"Dqn-tamer: Humanin-the-loop reinforcement learning with intractable feedback","authors":"Arakawa","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469- 483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bain and Sommut, 1999] Michael Bain and Claude Sommut. A framework for behavioural claning. Machine intelligence, 15(15):103, 1999.","title":"A framework for behavioural claning","authors":"Michael Bain, Claude Sommut","year":"1999","venue":"Machine intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bellemare et al., 2012] Marc G Bellemare, Yavar Naddaf, Joel Ve- ness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2012.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2012","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Busa-Fekete et al., 2013] Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference- based evolutionary direct policy search. In ICRA Workshop on Autonomous Learning, 2013.","title":"Preferencebased evolutionary direct policy search","authors":"Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, Eyke Hüllermeier","year":"2013","venue":"ICRA Workshop on Autonomous Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Cederborg et al., 2015] Thomas Cederborg, Ishaan Grover, Charles L Isbell, and Andrea L Thomaz. Policy shaping with hu- man teachers. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.","title":"Policy shaping with human teachers","authors":"Thomas Cederborg, Ishaan Grover, Charles L Isbell, Andrea L Thomaz","year":"2015","venue":"Twenty-Fourth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/474"},{"position":12,"raw":"[Christiano et al., 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299-4307, 2017.","title":"Deep reinforcement learning from human preferences","authors":"Jan Paul F Christiano, Tom Leike, Miljan Brown, Shane Martic, Dario Legg, Amodei","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dwibedi et al., 2018] Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable repre- sentations from visual observations. In 2018 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 1577-1584. IEEE, 2018.","title":"Learning actionable representations from visual observations","authors":"Dwibedi","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Edwards et al., 2018] Ashley D Edwards, Himanshu Sahni, Yan- nick Schroeker, and Charles L Isbell. Imitating latent policies from observation. arXiv preprint arXiv:1805.07914, 2018.","title":"Imitating latent policies from observation","authors":"Himanshu Ashley D Edwards, Yannick Sahni, Charles L Schroeker, Isbell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Fürnkranz et al., 2012] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based rein- forcement learning: a formal framework and a policy iteration algorithm. Machine learning, 89(1-2):123-156, 2012.","title":"Preference-based reinforcement learning: a formal framework and a policy iteration algorithm","authors":"Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, Sang-Hyeun Park","year":"2012","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Griffith et al., 2013] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in neural information processing systems, pages 2625-2633, 2013.","title":"Policy shaping: Integrating human feedback with reinforcement learning","authors":"Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, Andrea L Thomaz","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Guo et al., 2019] Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid reinforcement learning with expert state sequences. Association for the Advancement of Artificial Intelligence, 2019.","title":"Hybrid reinforcement learning with expert state sequences","authors":"Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, Murray Campbell","year":"2019","venue":"Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gupta et al., 2018] Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In Inter- national Conference on Learning Representations, 2018.","title":"Learning invariant feature spaces to transfer skills with reinforcement learning","authors":"Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Henderson et al., 2018] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Con- ference on Artificial Intelligence, 2018.","title":"Deep reinforcement learning that matters","authors":"Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Genera- tive adversarial imitation learning. In Advances in Neural Infor- mation Processing Systems, pages 4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ho et al., 2016] Mark K Ho, Michael Littman, James Mac- Glashan, Fiery Cushman, and Joseph L Austerweil. Showing versus doing: Teaching by demonstration. In Advances in neural information processing systems, pages 3027-3035, 2016.","title":"Showing versus doing: Teaching by demonstration","authors":"K Mark, Michael Ho, James Littman, Fiery Mac-Glashan, Joseph L Cushman, Austerweil","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hussein et al., 2017] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):21, 2017.","title":"Imitation learning: A survey of learning methods","authors":"Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, Chrisina Jayne","year":"2017","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ibarz et al., 2018] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, pages 8022-8034, 2018.","title":"Reward learning from human preferences and demonstrations in atari","authors":"Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Isbell et al., 2001] Charles Isbell, Christian R Shelton, Michael Kearns, Satinder Singh, and Peter Stone. A social reinforcement learning agent. In Proceedings of the fifth international confer- ence on Autonomous agents, pages 377-384. ACM, 2001.","title":"A social reinforcement learning agent","authors":"Charles Isbell, Michael Christian R Shelton, Satinder Kearns, Peter Singh, Stone","year":"2001","venue":"Proceedings of the fifth international conference on Autonomous agents","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kim et al., 2018] Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self- driving vehicles. In Proceedings of the European Conference on Computer Vision (ECCV), pages 563-578, 2018.","title":"Textual explanations for selfdriving vehicles","authors":"Kim","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Knox and Stone, 2009] W Bradley Knox and Peter Stone. Interac- tively shaping agents via human reinforcement: The tamer frame- work. In Proceedings of the fifth international conference on Knowledge capture, pages 9-16. ACM, 2009.","title":"Interactively shaping agents via human reinforcement: The tamer framework","authors":"Knox Bradley, Peter Stone","year":"2009","venue":"Proceedings of the fifth international conference on Knowledge capture","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Knox and Stone, 2010] W Bradley Knox and Peter Stone. Com- bining manual feedback with subsequent mdp reward signals for reinforcement learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: vol- ume 1-Volume 1, pages 5-12. International Foundation for Au- tonomous Agents and Multiagent Systems, 2010.","title":"Combining manual feedback with subsequent mdp reward signals for reinforcement learning","authors":"Knox Bradley, Peter Stone","year":"2010","venue":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Knox and Stone, 2012] W Bradley Knox and Peter Stone. Rein- forcement learning from simultaneous human and mdp reward. In Proceedings of the 11th International Conference on Au- tonomous Agents and Multiagent Systems-Volume 1, pages 475- 482. International Foundation for Autonomous Agents and Mul- tiagent Systems, 2012.","title":"Reinforcement learning from simultaneous human and mdp reward","authors":"Knox Bradley, Peter Stone","year":"2012","venue":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Le et al., 2018] Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé. Hierarchical imitation and reinforcement learning. In International Conference on Machine Learning, pages 2923-2932, 2018.","title":"Hierarchical imitation and reinforcement learning","authors":"Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, Hal Daumé","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Li et al., 2018] Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 619-635, 2018.","title":"In the eye of beholder: Joint learning of gaze and actions in first person video","authors":"Yin Li, Miao Liu, James M Rehg","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Liu et al., 2018] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1118-1125. IEEE, 2018.","title":"Imitation from observation: Learning to imitate behaviors from raw video via context translation","authors":"Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, Sergey Levine","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Loftin et al., 2016] Robert Loftin, Bei Peng, James MacGlashan, Michael L Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. Learning behaviors via human-delivered discrete feed- back: modeling implicit feedback strategies to speed up learning. Autonomous agents and multi-agent systems, 30(1):30-59, 2016.","title":"Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning","authors":"Robert Loftin, Bei Peng, James Macglashan, Matthew E Michael L Littman, Jeff Taylor, David L Huang, Roberts","year":"2016","venue":"Autonomous agents and multi-agent systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[MacGlashan et al., 2017] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Tay- lor, and Michael L Littman. Interactive learning from policy- dependent human feedback. In Proceedings of the 34th Interna- tional Conference on Machine Learning-Volume 70, pages 2285- 2294. JMLR. org, 2017.","title":"Interactive learning from policydependent human feedback","authors":"James Macglashan, K Mark, Robert Ho, Bei Loftin, Guan Peng, David L Wang, Matthew E Roberts, Taylor, Michael L Littman","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Merel et al., 2017] Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201, 2017.","title":"Learning human behaviors from motion capture by adversarial imitation","authors":"Merel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostro- vski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Nair et al., 2017] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Com- bining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2146-2153. IEEE, 2017.","title":"Combining self-supervised learning and imitation for vision-based rope manipulation","authors":"Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, Sergey Levine","year":"2017","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Osa et al., 2018] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorith- mic perspective on imitation learning. Foundations and Trends R in Robotics, 7(1-2):1-179, 2018.","title":"An algorithmic perspective on imitation learning","authors":"Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, Jan Peters","year":"2018","venue":"Foundations and Trends R in Robotics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Palazzi et al., 2018] Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, and Rita Cucchiara. Predicting the driver's focus of attention: the dr (eye) ve project. IEEE transac- tions on pattern analysis and machine intelligence, 2018.","title":"Predicting the driver's focus of attention: the dr (eye) ve project","authors":"Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, Rita Cucchiara","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Pilarski et al., 2011] Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton. Online human training of a myoelectric prosthesis con- troller via actor-critic reinforcement learning. In 2011 IEEE In- ternational Conference on Rehabilitation Robotics, pages 1-7. IEEE, 2011.","title":"Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning","authors":"Patrick M Pilarski, Thomas Michael R Dawson, Farbod Degris, Jason P Fahimi, Richard S Carey, Sutton","year":"2011","venue":"IEEE International Conference on Rehabilitation Robotics","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Pinsler et al., 2018] Robert Pinsler, Riad Akrour, Takayuki Osa, Jan Peters, and Gerhard Neumann. Sample and feedback efficient hierarchical reinforcement learning from human preferences. In 2018 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 596-601. IEEE, 2018.","title":"Sample and feedback efficient hierarchical reinforcement learning from human preferences","authors":"Robert Pinsler, Riad Akrour, Takayuki Osa, Jan Peters, Gerhard Neumann","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey J Gordon, Drew Bagnell","year":"2011","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Saunders et al., 2018] William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. In Proceed- ings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2067-2069. International Foun- dation for Autonomous Agents and Multiagent Systems, 2018.","title":"Trial without error: Towards safe reinforcement learning via human intervention","authors":"William Saunders, Girish Sastry, Andreas Stuhlmueller, Owain Evans","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Schaal, 1999] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233-242, 1999.","title":"Is imitation learning the route to humanoid robots","authors":"Stefan Schaal, Schaal","year":"1999","venue":"Trends in cognitive sciences","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Sermanet et al., 2018] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141. IEEE, 2018.","title":"Time-contrastive networks: Self-supervised learning from video","authors":"Sermanet","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural net- works and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Julian Driessche, Ioannis Schrittwieser, Veda Antonoglou, Marc Panneershelvam, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Stadie et al., 2017] Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. International Con- ference on Learning Representations, 2017.","title":"Third-person imitation learning","authors":"Pieter Bradly C Stadie, Ilya Abbeel, Sutskever","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for tempo- ral abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Tenorio-Gonzalez et al., 2010] Ana C Tenorio-Gonzalez, Ed- uardo F Morales, and Luis Villaseñor-Pineda. Dynamic reward shaping: training a robot by voice. In Ibero-American conference on artificial intelligence, pages 483-492. Springer, 2010.","title":"Dynamic reward shaping: training a robot by voice","authors":"Ana C Tenorio-Gonzalez, Eduardo F Morales, Luis Villaseñor-Pineda","year":"2010","venue":"Ibero-American conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Thomaz and Breazeal, 2008] Andrea L Thomaz and Cynthia Breazeal. Teachable robots: Understanding human teaching be- havior to build more effective robot learners. Artificial Intelli- gence, 172(6-7):716-737, 2008.","title":"Teachable robots: Understanding human teaching behavior to build more effective robot learners","authors":"L Andrea, Cynthia Thomaz, Breazeal","year":"2008","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Torabi et al., 2018a] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4950-4957. AAAI Press, 2018.","title":"Behavioral cloning from observation","authors":"Torabi","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/687"},{"position":54,"raw":"[Torabi et al., 2018b] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158, 2018.","title":"Generative adversarial imitation from observation","authors":"Torabi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Torabi et al., 2019a] Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial imitation learning from state-only demonstra- tions. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2229-2231. International Foundation for Autonomous Agents and Multiagent Systems, 2019.","title":"Adversarial imitation learning from state-only demonstrations","authors":"Torabi","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Torabi et al., 2019b] Faraz Torabi, Garrett Warnell, and Peter Stone. Imitation learning from video by leveraging propriocep- tion. arXiv preprint arXiv:1905.09335, 2019.","title":"Imitation learning from video by leveraging proprioception","authors":"Torabi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2019/497"},{"position":57,"raw":"[Torabi et al., 2019c] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. arXiv preprint arXiv:1905.13566, 2019.","title":"Recent advances in imitation learning from observation","authors":"Torabi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2019/882"},{"position":58,"raw":"[Warnell et al., 2018] Garrett Warnell, Nicholas Waytowich, Ver- non Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Deep tamer: Interactive agent shaping in high-dimensional state spaces","authors":"Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, Peter Stone","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Wilson et al., 2012] Aaron Wilson, Alan Fern, and Prasad Tade- palli. A bayesian approach for policy learning from trajectory preference queries. In Advances in neural information process- ing systems, pages 1133-1141, 2012.","title":"A bayesian approach for policy learning from trajectory preference queries","authors":"Aaron Wilson, Alan Fern, Prasad Tadepalli","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Wirth et al., 2016] Christian Wirth, Johannes Fürnkranz, and Ger- hard Neumann. Model-free preference-based reinforcement learning. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.","title":"Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-free preference-based reinforcement learning","authors":"Wirth","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Wirth et al., 2017] Christian Wirth, Riad Akrour, Gerhard Neu- mann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. The Journal of Machine Learn- ing Research, 18(1):4945-4990, 2017.","title":"Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods","authors":"Wirth","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Zhang et al., 2018] Ruohan Zhang, Zhuode Liu, Luxin Zhang, Jake A Whritner, Karl S Muller, Mary M Hayhoe, and Dana H Ballard. Agil: Learning attention from human for visuomotor tasks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 663-679, 2018.","title":"Agil: Learning attention from human for visuomotor tasks","authors":"Zhang","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Zhang et al., 2019] Ruohan Zhang, Zhuode Liu, Lin Guan, Luxin Zhang, Mary M Hayhoe, and Dana H Ballard. Atari-head: Atari human eye-tracking and demonstration dataset. arXiv preprint arXiv:1903.06754, 2019.","title":"Atari-head: Atari human eye-tracking and demonstration dataset","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Zintgraf et al., 2018] Luisa M Zintgraf, Diederik M Roijers, Sjo- erd Linders, Catholijn M Jonker, and Ann Nowé. Ordered prefer- ence elicitation strategies for supporting multi-objective decision making. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 1477-1485. International Foundation for Autonomous Agents and Multiagent Systems, 2018.","title":"Ordered preference elicitation strategies for supporting multi-objective decision making","authors":"Luisa M Zintgraf, M Diederik, Sjoerd Roijers, Catholijn M Linders, Ann Jonker, Nowé","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}