{"type":"ScholarlyArticle","title":"Improving Reinforcement Learning with Human Input","authors":[{"name":"Matthew E. Taylor","slug":"matthew-e-taylor","given":"Matthew E.","family":"Taylor","affiliation":"Borealis AI , Edmonton , AB"}],"abstract":"Reinforcement learning (RL) has had many successes when learning autonomously. This paper and accompanying talk consider how to make use of a non-technical human participant, when available. In particular, we consider the case where a human could 1) provide demonstrations of good behavior, 2) provide online evaluative feedback, or 3) define a curriculum of tasks for the agent to learn on. In all cases, our work has shown such information can be effectively leveraged. After giving a high-level overview of this work, we will highlight a set of open questions and suggest where future work could be usefully focused.","doi":"10.24963/ijcai.2018/817","url":"https://ijcai.org/proceedings/2018/817","pdf":"https://ijcai.org/proceedings/2018/0817.pdf","markdown":"https://ijcai.org/proceedings/2018/817.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/817","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Early Career","track":"Early Career","number":817,"pages":{"first":5724,"last":5728},"keywords":["Machine Learning: Reinforcement Learning","Humans and AI: Human-Computer Interaction","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Argall et al., 2009] Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robot. Auton. Syst., 57(5):469-483, May 2009.","title":"A survey of robot learning from demonstration","authors":"Brenna D Argall, Sonia Chernova, Manuela Veloso, Brett Browning","year":"2009","venue":"Robot. Auton. Syst","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boularias et al., 2011] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In AI Stats, pages 182-189, 2011.","title":"Relative entropy inverse reinforcement learning","authors":"Abdeslam Boularias, Jens Kober, Jan Peters","year":"2011","venue":"AI Stats","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 1990] Peter F Brown, John Cocke, Stephen A Della Pietra, Vincent J Della Pietra, Fredrick Jelinek, John D Lafferty, Robert L Mercer, and Paul S Roossin. A statistical approach to machine translation. Computational linguistics, 16(2):79-85, 1990.","title":"A statistical approach to machine translation","authors":"John Peter F Brown, Stephen A Cocke, Della Pietra, Vincent J Della Pietra, Fredrick Jelinek, John D Lafferty, Robert L Mercer, Paul S Roossin","year":"1990","venue":"Computational linguistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Harutyunyan et al., 2015] Anna Harutyunyan, Sam Devlin, Peter Vrancx, and Ann Nowe. Expressing arbitrary reward functions as potential-based advice. In AAAI, pages 2652-2658, 2015.","title":"Expressing arbitrary reward functions as potential-based advice","authors":"Anna Harutyunyan, Sam Devlin, Peter Vrancx, Ann Nowe","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Karakovskiy and Togelius, 2012] S. Karakovskiy and J. Togelius. The Mario AI benchmark and competitions. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):55-67, March 2012.","title":"The Mario AI benchmark and competitions","authors":"S Karakovskiy, J Togelius","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karpathy and Van De Panne, 2012] Andrej Karpathy and Michiel Van De Panne. Curriculum learning for motor skills. Advances in AI, pages 1015-1023, 2012.","title":"Curriculum learning for motor skills","authors":"Van Karpathy, Andrej De Panne, Michiel Karpathy, Van De Panne","year":"2012","venue":"Advances in AI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Knox and Stone, 2009] W. Bradley Knox and Peter Stone. Interac- tively Shaping Agents via Human Reinforcement: The TAMER Framework. In KCAP, pages 9-16, 2009.","title":"Interactively Shaping Agents via Human Reinforcement: The TAMER Framework","authors":"W, Bradley Knox, Peter Stone","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kumar et al., 2010] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Proc. of NIPS, pages 1189-1197, 2010.","title":"Self-paced learning for latent variable models","authors":"Benjamin M Pawan Kumar, Daphne Packer, Koller","year":"2010","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee and Grauman, 2011] Yong Jae Lee and Kristen Grauman. Learning the easy things first: Self-paced visual category dis- covery. In CVPR, pages 1721-1728, 2011.","title":"Learning the easy things first: Self-paced visual category discovery","authors":"Jae Yong, Kristen Lee, Grauman","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Guangliang Li, Shimon Whiteson, W. Bradley Knox, and Hayley Hung. Using informative behavior to increase engagement while learning from human reward. Autonomous Agents and Multi-Agent Systems, 30(5):826-848, 2016.","title":"Using informative behavior to increase engagement while learning from human reward","authors":"Guangliang Li, Shimon Whiteson, W Bradley Knox, Hayley Hung","year":"2016","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Loftin et al., 2015] Robert Loftin, Bei Peng, James MacGlashan, Michael L Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. Learning behaviors via human-delivered discrete feed- back: modeling implicit feedback strategies to speed up learn- ing. Autonomous Agents and Multi-Agent Systems, 30(1):30-59, 2015.","title":"Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning","authors":"Robert Loftin, Bei Peng, James Macglashan, Matthew E Michael L Littman, Jeff Taylor, David L Huang, Roberts","year":"2015","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[MacGlashan et al., 2014] J. MacGlashan, M. L. Littman, R. Loftin, B. Peng, D. L. Roberts, and M. E. Taylor. Training an agent to ground commands with reward and punishment. In Proceedings of the AAAI Machine Learning for Interactive Systems Workshop, pages 6-12, 2014.","title":"Training an agent to ground commands with reward and punishment","authors":"Macglashan","year":"2014","venue":"Proceedings of the AAAI Machine Learning for Interactive Systems Workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Narvekar et al., 2016] Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, and Peter Stone. Source task creation for curriculum learning. In AAMAS, pages 566-574, 2016.","title":"Source task creation for curriculum learning","authors":"Narvekar","year":"2016","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ng et al., 1999] Andrew Y. Ng, Daishi Harada, and Stuart J. Rus- sell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Y Ng, Daishi Harada, Stuart J Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peng et al., 2016] Bei Peng, James MacGlashan, Robert Loftin, Michael L. Littman, David L. Roberts, and Matthew E. Taylor. A Need for Speed: Adapting Agent Action Speed to Improve Task Learning from Non-Expert Humans. In Proc. of AAMAS, pages 957-965, 2016.","title":"A Need for Speed: Adapting Agent Action Speed to Improve Task Learning from Non-Expert Humans","authors":"Bei Peng, James Macglashan, Robert Loftin, Michael L Littman, David L Roberts, Matthew E Taylor","year":"2016","venue":"Proc. of AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rosenfeld et al., 2017] Ariel Rosenfeld, Matthew E. Taylor, and Sarit Kraus. Leveraging Human Knowledge in Tabular Rein- forcement Learning: A Study of Human Subjects. In Proc. of IJCAI, pages 3823-3830, 2017.","title":"Leveraging Human Knowledge in Tabular Reinforcement Learning: A Study of Human Subjects","authors":"Ariel Rosenfeld, Matthew E Taylor, Sarit Kraus","year":"2017","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/534"},{"position":18,"raw":"[Stanley et al., 2005] Kenneth O. Stanley, Bobby D. Bryant, and Risto Miikkulainen. Evolving neural network agents in the NERO video game. In CIG, pages 182-189, 2005.","title":"Evolving neural network agents in the NERO video game","authors":"Kenneth O Stanley, Bobby D Bryant, Risto Miikkulainen","year":"2005","venue":"CIG","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Stone et al., 2006] Peter Stone, Gregory Kuhlmann, Matthew E. Taylor, and Yaxin Liu. Keepaway Soccer: From Machine Learn- ing Testbed to Benchmark. In RoboCup-2005: Robot Soccer World Cup IX, Pages 93-105, 2006.","title":"Keepaway Soccer: From Machine Learning Testbed to Benchmark","authors":"Peter Stone, Gregory Kuhlmann, Matthew E Taylor, Yaxin Liu","year":"2006","venue":"RoboCup-2005: Robot Soccer World Cup IX","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Suay et al., 2016] Halit Bener Suay, Tim Brys, Matthew E. Taylor, and Sonia Chernova. Learning from Demonstration for Shaping through Inverse Reinforcement Learning. In Proc. of AAMAS, pages 429-437, 2016.","title":"Learning from Demonstration for Shaping through Inverse Reinforcement Learning","authors":"Halit Bener Suay, Tim Brys, Matthew E Taylor, Sonia Chernova","year":"2016","venue":"Proc. of AAMAS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] R.S. Sutton and A.G. Barto. Reinforce- ment learning: An introduction. MIT Press, 1998.","title":"Reinforcement learning: An introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Svetlik et al., 2016] M Svetlik, M Leonetti, J Sinapov, R Shah, N Walker, and P Stone. Automatic curriculum graph genera- tion for reinforcement learning agents. In Proc. of AAAI, pages 2590-2596, 2016.","title":"Automatic curriculum graph generation for reinforcement learning agents","authors":"M Svetlik, J Leonetti, R Sinapov, N Shah, Walker, Stone","year":"2016","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Taylor and Stone, 2007] Matthew E. Taylor and Peter Stone. Cross-Domain Transfer for Reinforcement Learning. In Proc. of ICML, pages 879-886, 2007.","title":"Cross-Domain Transfer for Reinforcement Learning","authors":"Matthew E Taylor, Peter Stone","year":"2007","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Taylor and Stone, 2009] Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(1):1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"Matthew E Taylor, Peter Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Taylor et al., 2007] Matthew E. Taylor, Peter Stone, and Yaxin Liu. Transfer Learning via Inter-Task Mappings for Temporal Difference Learning. Journal of Machine Learning Research, 8(1):2125-2167, 2007.","title":"Transfer Learning via Inter-Task Mappings for Temporal Difference Learning","authors":"Matthew E Taylor, Peter Stone, Yaxin Liu","year":"2007","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Taylor et al., 2011] Matthew E. Taylor, Halit Bener Suay, and So- nia Chernova. Integrating Reinforcement Learning with Human Demonstrations of Varying Ability. In Proc. of AAMAS, pages 617-624, 2011.","title":"Integrating Reinforcement Learning with Human Demonstrations of Varying Ability","authors":"Matthew E Taylor, Halit Bener Suay, Sonia Chernova","year":"2011","venue":"Proc. of AAMAS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Thomaz and Breazeal, 2006] Andrea Lockerd Thomaz and Cyn- thia Breazeal. Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learn- ing Performance. In AAAI, pages 1000-1005, 2006.","title":"Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learning Performance","authors":"Andrea Lockerd, Thomaz, Cynthia Breazeal","year":"2006","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Taylor, 2017] Zhaodong Wang and Matthew E. Tay- lor. Improving Reinforcement Learning with Confidence-Based Demonstrations. In IJCAI, pages 3027-3033, 2017.","title":"Improving Reinforcement Learning with Confidence-Based Demonstrations","authors":"Wang, Taylor ;, Zhaodong Wang, Matthew E Taylor","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/422"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}