{"type":"ScholarlyArticle","title":"Towards Sample Efficient Reinforcement Learning","authors":[{"name":"Yang Yu","slug":"yang-yu","given":"Yang","family":"Yu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"Reinforcement learning is a major tool to realize intelligent agents that can be autonomously adaptive to the environment. With deep models, reinforcement learning has shown great potential in complex tasks such as playing games from pixels. However, current reinforcement learning techniques are still suffer from requiring a huge amount of interaction data, which could result in unbearable cost in real-world applications. In this article, we share our understanding of the problem, and discuss possible ways to alleviate the sample cost of reinforcement learning, from the aspects of exploration, optimization, environment modeling, experience transfer, and abstraction. We also discuss some challenges in real-world applications, with the hope of inspiring future researches.","doi":"10.24963/ijcai.2018/820","url":"https://ijcai.org/proceedings/2018/820","pdf":"https://ijcai.org/proceedings/2018/0820.pdf","markdown":"https://ijcai.org/proceedings/2018/820.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/820","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Early Career","track":"Early Career","number":820,"pages":{"first":5739,"last":5743},"keywords":["Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option- critic architecture. In AAAI, 2017.","title":"The optioncritic architecture","authors":"Pierre-Luc Bacon, Jean Harb, Doina Precup","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies-A comprehensive introduction. Natural Com- puting, 1(1):3-52, 2002.","title":"Evolution strategies-A comprehensive introduction","authors":"Hans-Georg Beyer, Hans-Paul Schwefel","year":"2002","venue":"Natural Computing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Shi-Yong Chen, Yang Yu, Qing Da, et al. Stabilizing reinforce- ment learning in dynamic environment with application to online recommendation. In KDD, 2018.","title":"Stabilizing reinforcement learning in dynamic environment with application to online recommendation","authors":"Yang Shi-Yong Chen, Qing Yu, Da","year":"2018","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Wang-Zhou Dai, Qiu-Ling Xu, Yang Yu, , and Zhi-Hua Zhou. Tunneling neural perception and logic reasoning through abductive learning. arXiv:1802.01173, 2018.","title":"Tunneling neural perception and logic reasoning through abductive learning","authors":"Wang-Zhou Dai, Qiu-Ling Xu, Yang Yu, Zhi-Hua Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In NIPS, 1992.","title":"Feudal reinforcement learning","authors":"Peter Dayan, Geoffrey E Hinton","year":"1992","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, et al. Deep reinforcement learning in large discrete action spaces. arXiv:1512.07679, 2016.","title":"Deep reinforcement learning in large discrete action spaces","authors":"Gabriel Dulac-Arnold, Richard Evans, Hado Van Hasselt","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.","title":"Learning explanatory rules from noisy data","authors":"Richard Evans, Edward Grefenstette","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Eliseo Ferrante, Alessandro Lazaric, and Marcello Restelli. Transfer of task representation in reinforcement learning using policy-based proto-value functions. In AAMAS, 2008.","title":"Transfer of task representation in reinforcement learning using policy-based proto-value functions","authors":"Eliseo Ferrante, Alessandro Lazaric, Marcello Restelli","year":"2008","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model- agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.","title":"Modelagnostic meta-learning for fast adaptation of deep networks","authors":"Chelsea Finn, Pieter Abbeel, Sergey Levine","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In ICLR, 2017.","title":"Stochastic neural networks for hierarchical reinforcement learning","authors":"Carlos Florensa, Yan Duan, Pieter Abbeel","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, et al. Noisy networks for exploration. In ICLR, 2018.","title":"Noisy networks for exploration","authors":"Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Alex Graves, Greg Wayne, Malcolm Reynolds, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016.","title":"Hybrid computing using a neural network with dynamic external memory","authors":"Alex Graves, Greg Wayne, Malcolm Reynolds","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. arXiv:1603.06318, 2016.","title":"Harnessing deep neural networks with logic rules","authors":"Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, Eric Xing","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Yi-Qi Hu, Hong Qian, and Yang Yu. Sequential classification- based optimization for direct policy search. In AAAI, 2017.","title":"Sequential classificationbased optimization for direct policy search","authors":"Yi-Qi Hu, Hong Qian, Yang Yu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Yujing Hu, Qing Da, Anxiang Zeng, Yang Yu, and Yinghui Xu. Reinforcement learning to rank in e-commerce search engine: Formalization, analysis, and application. In KDD, 2018.","title":"Reinforcement learning to rank in e-commerce search engine: Formalization, analysis, and application","authors":"Yujing Hu, Qing Da, Anxiang Zeng, Yang Yu, Yinghui Xu","year":"2018","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Max Jaderberg, Valentin Dalibard, Simon Osindero, et al. Population based training of neural networks. arXiv:1711.09846, 2017.","title":"Population based training of neural networks","authors":"Max Jaderberg, Valentin Dalibard, Simon Osindero","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples in batch reinforcement learning. In ICML, 2008.","title":"Transfer of samples in batch reinforcement learning","authors":"Alessandro Lazaric, Marcello Restelli, Andrea Bonarini","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Ronald Parr and Stuart J. Russell. Reinforcement learning with hierarchies of machines. In NIPS, 1997.","title":"Reinforcement learning with hierarchies of machines","authors":"Ronald Parr, Stuart J Russell","year":"1997","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. arXiv:1710.06537, 2018.","title":"Sim-to-real transfer of robotic control with dynamics randomization","authors":"Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforce- ment learning. arXiv:1712.06567, 2018.","title":"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning","authors":null,"year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, et al. Parameter space noise for exploration. In ICLR, 2018.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep rl for model- based control. In ICLR, 2018.","title":"Temporal difference models: Model-free deep rl for modelbased control","authors":"Vitchyr Pong, Shixiang Gu, Murtaza Dalal, Sergey Levine","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"Hong Qian, Yi-Qi Hu, and Yang Yu. Derivative-free optimiza- tion of high-dimensional non-convex functions by sequen- tial random embeddings. In IJCAI, 2016.","title":"Derivative-free optimization of high-dimensional non-convex functions by sequential random embeddings","authors":"Hong Qian, Yi-Qi Hu, Yang Yu","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/278"},{"position":27,"raw":"Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dy- namics: a nonasymptotic analysis. In COLT, 2017.","title":"Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis","authors":"Maxim Raginsky, Alexander Rakhlin, Matus Telgarsky","year":"2017","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Himanshu Sahni, Saurabh Kumar, Farhan Tejani, and Charles Isbell. Learning to compose skills. arXiv:1711.11289, 2017.","title":"Learning to compose skills","authors":"Himanshu Sahni, Saurabh Kumar, Farhan Tejani, Charles Isbell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jor- dan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"Jing-Cheng Shi, Yang Yu, Qing Da, Shi-Yong Chen, and An-Xiang Zeng. Virtual-Taobao: Virtualizing real- world online retail environment for reinforcement learning. arXiv:1805.10000, 2018.","title":"Virtual-Taobao: Virtualizing realworld online retail environment for reinforcement learning","authors":"Jing-Cheng Shi, Yang Yu, Qing Da, Shi-Yong Chen, An-Xiang Zeng","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"David Silver, Aja Huang, Chris J. Maddison, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"Satinder Singh, Andrew G. Barto, and Nuttapong Chentanez. Intrinsically motivated reinforcement learning. In NIPS, 2004.","title":"Intrinsically motivated reinforcement learning","authors":"Satinder Singh, Andrew G Barto, Nuttapong Chentanez","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NIPS, 2012.","title":"Practical bayesian optimization of machine learning algorithms","authors":"Jasper Snoek, Hugo Larochelle, Ryan P Adams","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"Richard S. Sutton and Andrew G. Barto. Reinforcement Learn- ing: An Introduction. MIT Press, Cambridge, MA, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"Richard S. Sutton, Doina Precup, and Satinder P. Singh. Be- tween MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Richard S Sutton, Doina Precup, Satinder P Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In NIPS, 2016.","title":"Value iteration networks","authors":"Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, Pieter Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, et al. Feudal networks for hierarchical reinforcement learn- ing. In ICML, 2017.","title":"Feudal networks for hierarchical reinforcement learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Schaul","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"Hong Wang, Hong Qian, and Yang Yu. Noisy derivative-free optimization with value suppression. In AAAI, 2018.","title":"Noisy derivative-free optimization with value suppression","authors":"Hong Wang, Hong Qian, Yang Yu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"Théophane Weber, Sébastien Racanière, David P. Reichert, et al. Imagination-augmented agents for deep reinforcement learning. arXiv:1707.06203, 2018.","title":"Imagination-augmented agents for deep reinforcement learning","authors":"Théophane Weber, Sébastien Racanière, David P Reichert","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"Shimon Whiteson. Evolutionary computation for reinforce- ment learning. In Marco Wiering and Martijn van Otterlo, editors, Reinforcement Learning: State-of-the-Art, pages 325-355. Springer, Berlin, Heidelberg, 2012.","title":"Evolutionary computation for reinforcement learning","authors":"Shimon Whiteson","year":"2012","venue":"Reinforcement Learning: State-of-the-Art","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"Yang Yu, Shi-Yong Chen, Qing Da, and Zhi-Hua Zhou. Reusable reinforcement learning via shallow trails, 2018.","title":"Reusable reinforcement learning via shallow trails","authors":"Yang Yu, Shi-Yong Chen, Qing Da, Zhi-Hua Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"Chao Zhang, Yang Yu, and Zhi-Hua Zhou. Learning envi- ronmental calibration actions for policy self-evolution. In IJCAI, 2018.","title":"Learning environmental calibration actions for policy self-evolution","authors":"Chao Zhang, Yang Yu, Zhi-Hua Zhou","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/425"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}