{"type":"ScholarlyArticle","title":"Potential Driven Reinforcement Learning for Hard Exploration Tasks","authors":[{"name":"Enmin Zhao","slug":"enmin-zhao","given":"Enmin","family":"Zhao","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhaoenmin2018 ,"},{"name":"Shihong Deng","slug":"shihong-deng","given":"Shihong","family":"Deng","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Yifan Zang","slug":"yifan-zang","given":"Yifan","family":"Zang","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhaoenmin2018 ,"},{"name":"Yongxin Kang","slug":"yongxin-kang","given":"Yongxin","family":"Kang","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Sciences {zhaoenmin2018 ,; Institute of Automation , Chinese Academy of Sciences"},{"name":"Kai Li","slug":"kai-li","given":"Kai","family":"Li","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Junliang Xing","slug":"junliang-xing","given":"Junliang","family":"Xing","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhaoenmin2018 ,"}],"abstract":"Experience replay plays a crucial role in Reinforcement Learning (RL), enabling the agent to remember and reuse experience from the past. Most previous methods sample experience transitions using simple heuristics like uniformly sampling or prioritizing those good ones. Since humans can learn from both good and bad experiences, more sophisticated experience replay algorithms need to be developed. Inspired by the potential energy in physics, this work introduces the artificial potential field into experience replay and develops Potentialized Experience Replay (PotER) as a new and effective sampling algorithm for RL in hard exploration tasks with sparse rewards. PotER defines a potential energy function for each state in experience replay and helps the agent to learn from both good and bad experiences using intrinsic state supervision. PotER can be combined with different RL algorithms as well as the self-imitation learning algorithm. Experimental analyses and comparisons on multiple challenging hard exploration environments have verified its effectiveness and efficiency.","doi":"10.24963/ijcai.2020/290","url":"https://ijcai.org/proceedings/2020/290","pdf":"https://ijcai.org/proceedings/2020/0290.pdf","markdown":"https://ijcai.org/proceedings/2020/290.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/290","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":290,"pages":{"first":2096,"last":2102},"keywords":["Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Combinatorial Search and Optimisation","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24673/file_24673.m3u8","length_sec":264,"provider":"hls"}],"references":[{"position":1,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048-5058, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mcgrew, Josh Tobin, Openai, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aytar et al., 2018] Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pages 2935-2945, 2018.","title":"Playing hard exploration games by watching youtube","authors":"Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, Nando De Freitas","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Uni- fying count-based exploration and intrinsic motivation. In Ad- vances in Neural Information Processing Systems, pages 1471- 1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Interna- tional Conference on Machine Learning, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burda et al., 2019] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.","title":"Exploration by random network distillation","authors":"Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dorigo and Birattari, 2010] Marco Dorigo and Mauro Birattari. Ant colony optimization. Springer, 2010.","title":"Ant colony optimization","authors":"Marco Dorigo, Mauro Birattari","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gruslys et al., 2018] Audrunas Gruslys, Mohammad Gheshlaghi Azar, Marc G Bellemare, and Remi Munos. The reactor: A sample-efficient actor-critic architecture. International Confer- ence on Learning Representations, 2018.","title":"The reactor: A sample-efficient actor-critic architecture","authors":"Gruslys","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hester et al., 2018] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep Q-learning from demonstrations. In AAAI Conference on Artificial Intelligence, pages 3223-3230, 2018.","title":"Deep Q-learning from demonstrations","authors":"Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khatib, 1986] Oussama Khatib. Real-time obstacle avoidance for manipulators and mobile robots. In Autonomous Robot Vehicles, pages 396-404, 1986.","title":"Real-time obstacle avoidance for manipulators and mobile robots","authors":"Oussama Khatib","year":"1986","venue":"Autonomous Robot Vehicles","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2019] Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. In International Conference on Ma- chine Learning, pages 3360-3369, 2019.","title":"Emi: Exploration with mutual information","authors":"Kim","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Le et al., 2018] Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé, III. Hierarchical imitation and reinforcement learning. In International Conference on Ma- chine Learning, pages 2917-2926, 2018.","title":"Hierarchical imitation and reinforcement learning","authors":"Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, Hal Daumé, Iii","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. In International Conference on Learning Representations, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Machado et al., 2018] Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. In International Conference on Learning Representations, 2018.","title":"Eigenoption discovery through the deep successor representation","authors":"C Marlos, Clemens Machado, Xiaoxiao Rosenbaum, Miao Guo, Gerald Liu, Murray Tesauro, Campbell","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostro- vski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nair et al., 2018] Ashvin Nair, Bob McGrew, Marcin Andrychow- icz, Wojciech Zaremba, and Pieter Abbeel. Overcoming explo- ration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation, pages 6292-6299, 2018.","title":"Overcoming exploration in reinforcement learning with demonstrations","authors":"Ashvin Nair, Bob Mcgrew, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel","year":"2018","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oh et al., 2018] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Confer- ence on Machine Learning, pages 3878-3887, 2018.","title":"Self-imitation learning","authors":"Junhyuk Oh, Yijie Guo, Satinder Singh, Honglak Lee","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In International Conference on Machine Learning, pages 2778-2787, 2017.","title":"Curiosity-driven exploration by selfsupervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pohlen et al., 2018] Tobias Pohlen, Bilal Piot, Todd Hester, Mo- hammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv:1805.11593, 2018.","title":"Observe and look further: Achieving consistent performance on atari","authors":"Pohlen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2019] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. In Advances in Neural Information Processing Systems, pages 13485-13496, 2019.","title":"Exploration via hindsight goal generation","authors":"Ren","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rimon and Koditschek, 1992] Elon Rimon and Daniel E Koditschek. Exact robot navigation using artificial poten- tial functions. In International Conference on Robotics and Automation, pages 501-518, 1992.","title":"Exact robot navigation using artificial potential functions","authors":"Koditschek Rimon, Elon Rimon, Daniel E Koditschek","year":"1992","venue":"International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Savinov et al., 2019] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In International Conference on Learning Representations, 2019.","title":"Episodic curiosity through reachability","authors":"Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy op- timization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stadie et al., 2015] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. In Advances in Neural Information Pro- cessing Systems Workshop, 2015.","title":"Incentivizing exploration in reinforcement learning with deep predictive models","authors":"Sergey Bradly C Stadie, Pieter Levine, Abbeel","year":"2015","venue":"Advances in Neural Information Processing Systems Workshop","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Strehl and Littman, 2008] Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sci- ences, 74(8):1309-1331, 2008.","title":"An analysis of model-based interval estimation for markov decision processes","authors":"L Alexander, Strehl, Michael L Littman","year":"2008","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. Exploration: A study of count- based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 2753-2762, 2017.","title":"Exploration: A study of countbased exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Openai Xi Chen, Yan Duan, John Schulman, Filip Deturck, Pieter Abbeel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2017] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In International Conference on Learning Representations, 2017.","title":"Sample efficient actor-critic with experience replay","authors":"Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, Nando De Freitas","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao and Tresp, 2018] Rui Zhao and Volker Tresp. Energy-based hindsight experience prioritization. In Conference on Robot Learning, pages 113-122, 2018.","title":"Energy-based hindsight experience prioritization","authors":"Rui Zhao, Volker Tresp","year":"2018","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}