{"type":"ScholarlyArticle","title":"Improving Reinforcement Learning with Confidence-Based Demonstrations","authors":[{"name":"Zhaodong Wang","slug":"zhaodong-wang","given":"Zhaodong","family":"Wang","affiliation":"School of EECS Washington State University"},{"name":"Matthew E. Taylor","slug":"matthew-e-taylor","given":"Matthew E.","family":"Taylor","affiliation":"School of EECS Washington State University"}],"abstract":"Reinforcement learning has had many successes, but in practice it often requires significant amounts of data to learn high-performing policies. One common way to improve learning is to allow a trained (source) agent to assist a new (target) agent. The goals in this setting are to 1) improve the target agent's performance, relative to learning unaided, and 2) allow the target agent to outperform the source agent. Our approach leverages source agent demonstrations, removing any requirements on the source agent's learning algorithm or representation. The target agent then estimates the source agent's policy and improves upon it. The key contribution of this work is to show that leveraging the target agent's uncertainty in the source agent's policy can significantly improve learning in two complex simulated domains, Keepaway and Mario.","doi":"10.24963/ijcai.2017/422","url":"https://ijcai.org/proceedings/2017/422","pdf":"https://ijcai.org/proceedings/2017/0422.pdf","markdown":"https://ijcai.org/proceedings/2017/422.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/422","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":422,"pages":{"first":3027,"last":3033},"keywords":["Multidisciplinary Topics and Applications: Human-Computer Interaction","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Albus, 1981] JS Albus. Brains, behavior. & Robotics. Pe- terboro, NH: Byte Books, 1981.","title":"Brains, behavior. & Robotics. Peterboro","authors":"Js Albus","year":"1981","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern recognition. Machine Learning, 128:1-58, 2006.","title":"Pattern recognition","authors":"M Christopher, Bishop","year":"2006","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Celeux and Govaert, 1992] Gilles Celeux and Gérard Gov- aert. A classification em algorithm for clustering and two stochastic versions. Computational statistics & Data anal- ysis, 14(3):315-332, 1992.","title":"Gilles Celeux and Gérard Govaert. A classification em algorithm for clustering and two stochastic versions","authors":"Govaert Celeux","year":"1992","venue":"Computational statistics & Data analysis","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chernova and Veloso, 2009] Sonia Chernova and Manuela Veloso. Interactive policy learning through confidence- based autonomy. Journal of Artificial Intelligence Re- search, 34(1):1, 2009.","title":"Interactive policy learning through confidencebased autonomy","authors":"Sonia Chernova, Manuela Veloso","year":"2009","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dietterich, 2000] Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pages 1-15. Springer, 2000.","title":"Ensemble methods in machine learning","authors":"G Thomas, Dietterich","year":"2000","venue":"International workshop on multiple classifier systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karakovskiy and Togelius, 2012] Sergey Karakovskiy and Julian Togelius. The mario ai benchmark and competi- tions. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):55-67, 2012.","title":"The mario ai benchmark and competitions","authors":"Sergey Karakovskiy, Julian Togelius","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin, 1992] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321, 1992.","title":"Self-improving reactive agents based on reinforcement learning, planning and teaching","authors":"Long-Ji Lin","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Noda et al., 1998] Itsuki Noda, Hitoshi Matsubara, Kazuo Hiraki, and Ian Frank. Soccer server: A tool for research on multiagent systems. Applied Artificial Intelligence, 12(2-3):233-250, 1998.","title":"Soccer server: A tool for research on multiagent systems","authors":"Itsuki Noda, Hitoshi Matsubara, Kazuo Hiraki, Ian Frank","year":"1998","venue":"Applied Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Quinlan, 1993] Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo, CA, 1993.","title":"C4.5: Programs for Machine Learning","authors":"Ross Quinlan","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rummery and Niranjan, 1994] Gavin A Rummery and Ma- hesan Niranjan. On-line q-learning using connectionist systems. 1994.","title":"On-line q-learning using connectionist systems","authors":"A Gavin, Mahesan Rummery, Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Singh and Sutton, 1996] Satinder P Singh and Richard S Sutton. Reinforcement learning with replacing eligibility traces. Machine learning, 22(1-3):123-158, 1996.","title":"Reinforcement learning with replacing eligibility traces","authors":"P Satinder, Richard S Singh, Sutton","year":"1996","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Stone et al., 2006] Peter Stone, Gregory Kuhlmann, Matthew E Taylor, and Yaxin Liu. Keepaway soc- cer: From machine learning testbed to benchmark. In RoboCup 2005: Robot Soccer World Cup IX, pages 93-105. Springer, 2006.","title":"Keepaway soccer: From machine learning testbed to benchmark","authors":"Peter Stone, Gregory Kuhlmann, Matthew E Taylor, Yaxin Liu","year":"2006","venue":"RoboCup 2005: Robot Soccer World Cup IX","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Suay et al., 2016] Halit Bener Suay, Tim Brys, Matthew E Taylor, and Sonia Chernova. Learning from demonstra- tion for shaping through inverse reinforcement learning. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pages 429- 437. International Foundation for Autonomous Agents and Multiagent Systems, 2016.","title":"Learning from demonstration for shaping through inverse reinforcement learning","authors":"Halit Bener Suay, Tim Brys, Matthew E Taylor, Sonia Chernova","year":"2016","venue":"Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"Barto ; Sutton, S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Taylor and Stone, 2007] Matthew E Taylor and Peter Stone. Cross-domain transfer for reinforcement learning. In Pro- ceedings of the 24th international conference on Machine learning, pages 879-886. ACM, 2007.","title":"Cross-domain transfer for reinforcement learning","authors":"E Matthew, Peter Taylor, Stone","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Taylor and Stone, 2009] Matthew E. Taylor and Peter Stone. Transfer Learning for Reinforcement Learning Do- mains: A Survey. Journal of Machine Learning Research, 10(1):1633-1685, 2009.","title":"Transfer Learning for Reinforcement Learning Domains: A Survey","authors":"Matthew E Taylor, Peter Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Taylor et al., 2011] Matthew E. Taylor, Halit Bener Suay, and Sonia Chernova. Integrating reinforcement learning with human demonstrations of varying ability. In Pro- ceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2011.","title":"Integrating reinforcement learning with human demonstrations of varying ability","authors":"Matthew E Taylor, Halit Bener Suay, Sonia Chernova","year":"2011","venue":"Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Witten and Frank, 2005] Ian H. Witten and Eibe Frank. Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco, 2nd edi- tion, 2005.","title":"Data Mining: Practical Machine Learning Tools and Techniques","authors":"Ian H Witten, Eibe Frank","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}