{"type":"ScholarlyArticle","title":"Interactive Reinforcement Learning with Dynamic Reuse of Prior Knowledge from Human and Agent Demonstrations","authors":[{"name":"Zhaodong Wang","slug":"zhaodong-wang","given":"Zhaodong","family":"Wang","affiliation":"School of EECS , Washington State University"},{"name":"Matthew E. Taylor","slug":"matthew-e-taylor","given":"Matthew E.","family":"Taylor","affiliation":"School of EECS , Washington State University"}],"abstract":"Reinforcement learning has enjoyed multiple impressive successes in recent years. However, these successes typically require very large amounts of data before an agent achieves acceptable performance. This paper focuses on a novel way of combating such requirements by leveraging existing (human or agent) knowledge. In particular, this paper leverages demonstrations, allowing an agent to quickly achieve high performance. This paper introduces the Dynamic Reuse of Prior (DRoP) algorithm, which combines the offline knowledge (demonstrations recorded before learning) with online confidence-based performance analysis. DRoP leverages the demonstrator's knowledge by automatically balancing between reusing the prior knowledge and the current learned policy, allowing the agent to outperform the original demonstrations. We compare with multiple state-of-the-art learning algorithms and empirically show that DRoP can achieve superior performance in two domains. Additionally, we show that this confidence measure can be used to selectively request additional demonstrations, significantly improving the learning performance of the agent.","doi":"10.24963/ijcai.2019/530","url":"https://ijcai.org/proceedings/2019/530","pdf":"https://ijcai.org/proceedings/2019/0530.pdf","markdown":"https://ijcai.org/proceedings/2019/530.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/530","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":530,"pages":{"first":3820,"last":3827},"keywords":["Machine Learning: Reinforcement Learning","Humans and AI: Human-AI Collaboration","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"Method Cartpole Mario Jumpstart Total Reward Final Reward jumpstart Total Reward Final","title":"Method Cartpole Mario Jumpstart Total Reward Final Reward jumpstart Total Reward Final","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Reward","title":"Reward","authors":null,"year":null,"venue":"Reward","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Q-Learning N/A 11653 951 ± 36 N/A 27141 1569 ± 51 HAT 201 20004 1444 ± 52 651 28828 1577 ± 41 CHAT 258 22692 1766 ± 68 1046 30144 1574 ± 46 Dagger 144 25728 1651 ± 62 437 25828 1526 ± 44","title":"Q-Learning N/A","authors":null,"year":null,"venue":"Q-Learning N/A","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"DRoP (DCU, Gaussian) 315 34025 2369 ± 72 923 32562 1699 ± 49","title":"DRoP (DCU, Gaussian)","authors":null,"year":null,"venue":"DRoP (DCU, Gaussian)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"DRoP (DRU, Gaussian) 320 35187 2395 ± 64 931 31241 1732 ± 46","title":"Gaussian)","authors":"( Drop, Dru","year":null,"venue":"Gaussian)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"DRoP (DCU, NN) 308 35312 2383 ± 71 909 32108 1752 ± 55","title":"DRoP","authors":null,"year":null,"venue":"DRoP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"DRoP (DRU, NN) 303 35544 2411 ± 56 915 33022 1779 ±","title":"NN","authors":"( Drop, Dru","year":null,"venue":"NN","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Albus, 1981] JS Albus. Brains, behavior. & Robotics. Pe- terboro, NH: Byte Books, 1981.","title":"Brains, behavior. & Robotics. Peterboro","authors":"Js Albus","year":"1981","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern recognition. Machine Learning, 128:1-58, 2006.","title":"Pattern recognition","authors":"M Christopher, Bishop","year":"2006","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016.","title":"OpenAI gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Brys et al., 2015] Tim Brys, Anna Harutyunyan, Matthew E Taylor, and Ann Nowé. Policy transfer using reward shap- ing. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 181-188, 2015.","title":"Policy transfer using reward shaping","authors":"Tim Brys, Anna Harutyunyan, Matthew E Taylor, Ann Nowé","year":"2015","venue":"Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Brys et al., 2017] Tim Brys, Anna Harutyunyan, Peter Vrancx, Ann Nowé, and Matthew E Taylor. Multi- objectivization and ensembles of shapings in reinforce- ment learning. Neurocomputing, 2017.","title":"Multiobjectivization and ensembles of shapings in reinforcement learning","authors":"Tim Brys, Anna Harutyunyan, Peter Vrancx, Ann Nowé, Matthew E Taylor","year":"2017","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Chernova and Veloso, 2007] Sonia Chernova and Manuela Veloso. Confidence-based policy learning from demon- stration using gaussian mixture models. In Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, 2007.","title":"Confidence-based policy learning from demonstration using gaussian mixture models","authors":"Veloso Chernova, Sonia Chernova, Manuela Veloso","year":"2007","venue":"Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Da Silva and Mackworth, 2010] Bruno N Da Silva and Alan Mackworth. Using spatial hints to improve policy reuse in a reinforcement learning agent. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, pages 317-324, 2010.","title":"Using spatial hints to improve policy reuse in a reinforcement learning agent","authors":"Da Silva, Mackworth ; Bruno N Da Silva, Alan Mackworth","year":"2010","venue":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Fernández and Veloso, 2006] Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in a rein- forcement learning agent. In Proceedings of the fifth International Joint Conference on Autonomous Agents and Multiagent Systems, pages 720-727. ACM, 2006.","title":"Probabilistic policy reuse in a reinforcement learning agent","authors":"Fernando Fernández, Manuela Veloso","year":"2006","venue":"Proceedings of the fifth International Joint Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Higgins et al., 2017] Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexan- der Lerchner. Darla: Improving zero-shot transfer in rein- forcement learning. In Thirty-fourth International Confer- ence on Machine Learning, 2017.","title":"Darla: Improving zero-shot transfer in reinforcement learning","authors":"Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, P Christopher, Alexander Burgess, Matthew Pritzel, Charles Botvinick, Alexander Blundell, Lerchner","year":"2017","venue":"Thirty-fourth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Karakovskiy and Togelius, 2012] Sergey Karakovskiy and Julian Togelius. The Mario AI benchmark and competi- tions. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):55-67, 2012.","title":"Sergey Karakovskiy and Julian Togelius. The Mario AI benchmark and competitions","authors":"Togelius Karakovskiy","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceed- ings of the National Academy of Sciences, 114(13):3521- 3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luo et al., 2017] Zelun Luo, Yuliang Zou, Judy Hoffman, and Li F Fei-Fei. Label efficient learning of transferable representations acrosss domains and tasks. In Advances in Neural Information Processing Systems, pages 164-176, 2017.","title":"Label efficient learning of transferable representations acrosss domains and tasks","authors":"Luo","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mao et al., 2018] Li Mao, Wei Yi, and Kudenko Daniel. In Adaptive and Learning Agents Workshop at AAMAS-18, 2018.","title":"Adaptive and Learning Agents Workshop at AAMAS-18","authors":"Li Mao, Wei Yi, Kudenko Daniel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Maurer et al., 2016] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. The Journal of Machine Learning Research, 17(1):2853-2884, 2016.","title":"The benefit of multitask representation learning","authors":"Andreas Maurer, Massimiliano Pontil, Bernardino Romera-Paredes","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algo- rithms for inverse reinforcement learning. In ICML, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Parisotto et al., 2016] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations (ICLR), 2016.","title":"Actor-mimic: Deep multitask and transfer reinforcement learning","authors":"Parisotto","year":"2016","venue":"Proceedings of the 4th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Quinlan, 1993] Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo, CA, 1993.","title":"Ross Quinlan. C4.5: Programs for Machine Learning","authors":"Quinlan","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceed- ings of the fourteenth International Conference on Artifi- cial Intelligence and Statistics, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey Gordon, Drew Bagnell","year":"2011","venue":"Proceedings of the fourteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rusu et al., 2016] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Had- sell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.","title":"Progressive neural networks","authors":"Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, Raia Hadsell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schaal, 1997] Stefan Schaal. Learning from demonstration. In Advances in neural information processing systems, pages 1040-1046, 1997.","title":"Learning from demonstration","authors":"Stefan Schaal, Schaal","year":"1997","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Suay et al., 2016] Halit Bener Suay, Tim Brys, Matthew E Taylor, and Sonia Chernova. Learning from demonstra- tion for shaping through inverse reinforcement learning. In Proceedings of the 2016 International Conference on Au- tonomous Agents & Multiagent Systems, pages 429-437, 2016.","title":"Learning from demonstration for shaping through inverse reinforcement learning","authors":"Halit Bener Suay, Tim Brys, Matthew E Taylor, Sonia Chernova","year":"2016","venue":"Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Taylor and Stone, 2007] Matthew E Taylor and Peter Stone. Cross-domain transfer for reinforcement learning. In Pro- ceedings of the 24th International Conference on Machine Learning, pages 879-886. ACM, 2007.","title":"Cross-domain transfer for reinforcement learning","authors":"E Matthew, Peter Taylor, Stone","year":"2007","venue":"Proceedings of the 24th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Taylor and Stone, 2009] Matthew E. Taylor and Peter Stone. Transfer Learning for Reinforcement Learning Do- mains: A Survey. Journal of Machine Learning Research, 10(1):1633-1685, 2009.","title":"Transfer Learning for Reinforcement Learning Domains: A Survey","authors":"Matthew E Taylor, Peter Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Taylor et al., 2011] Matthew E. Taylor, Halit Bener Suay, and Sonia Chernova. Integrating reinforcement learning with human demonstrations of varying ability. In Pro- ceedings of the International Conference on Autonomous Agents and Multiagent Systems, May 2011.","title":"Integrating reinforcement learning with human demonstrations of varying ability","authors":"Matthew E Taylor, Halit Bener Suay, Sonia Chernova","year":"2011","venue":"Proceedings of the International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Teh et al., 2017] Yee Teh, Victor Bapst, Wojciech M Czar- necki, John Quan, James Kirkpatrick, Raia Hadsell, Nico- las Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Informa- tion Processing Systems, pages 4499-4509, 2017.","title":"Distral: Robust multitask reinforcement learning","authors":"Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, Razvan Pascanu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang and Taylor, 2017] Zhaodong Wang and Matthew E. Taylor. Improving Reinforcement Learning with Confidence-Based Demonstrations. In Proceedings of the 26th International Conference on Artificial Intelligence (IJCAI), August 2017.","title":"Improving Reinforcement Learning with Confidence-Based Demonstrations","authors":"Wang, Taylor ;, Zhaodong Wang, Matthew E Taylor","year":"2017","venue":"Proceedings of the 26th International Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2017/422"},{"position":36,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Dayan ; Watkins, Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}