{"type":"ScholarlyArticle","title":"Reward Prediction Error as an Exploration Objective in Deep RL","authors":[{"name":"Riley Simmons-Edler","slug":"riley-simmons-edler","given":"Riley","family":"Simmons-Edler","affiliation":"Princeton University; Samsung AI Center NYC"},{"name":"Ben Eisner","slug":"ben-eisner","given":"Ben","family":"Eisner","affiliation":"Samsung AI Center NYC"},{"name":"Daniel Yang","slug":"daniel-yang","given":"Daniel","family":"Yang","affiliation":"Samsung AI Center NYC"},{"name":"Anthony Bisulco","slug":"anthony-bisulco","given":"Anthony","family":"Bisulco","affiliation":"Samsung AI Center NYC"},{"name":"Eric Mitchell","slug":"eric-mitchell","given":"Eric","family":"Mitchell","affiliation":"Samsung AI Center NYC; Stanford University"},{"name":"Sebastian Seung","slug":"sebastian-seung","given":"Sebastian","family":"Seung","affiliation":"Samsung AI Center NYC"},{"name":"Daniel Lee","slug":"daniel-lee","given":"Daniel","family":"Lee","affiliation":"Samsung AI Center NYC"}],"abstract":"A major challenge in reinforcement learning is exploration, when local dithering methods such as epsilon-greedy sampling are insufficient to solve a given task. Many recent methods have proposed to intrinsically motivate an agent to seek novel states, driving the agent to discover improved reward. However, while state-novelty exploration methods are suitable for tasks where novel observations correlate well with improved reward, they may not explore more efficiently than epsilon-greedy approaches in environments where the two are not well-correlated. In this paper, we distinguish between exploration tasks in which seeking novel states aids in finding new reward, and those where it does not, such as goal-conditioned tasks and escaping local reward maxima. We propose a new exploration objective, maximizing the reward prediction error (RPE) of a value function trained to predict extrinsic reward. We then propose a deep reinforcement learning method, QXplore, which exploits the temporal difference error of a Q-function to solve hard exploration tasks in high-dimensional MDPs. We demonstrate the exploration behavior of QXplore on several OpenAI Gym MuJoCo tasks and Atari games and observe that QXplore is comparable to or better than a baseline state-novelty method in all cases, outperforming the baseline on tasks where state novelty is not well-correlated with improved reward.","doi":"10.24963/ijcai.2020/390","url":"https://ijcai.org/proceedings/2020/390","pdf":"https://ijcai.org/proceedings/2020/0390.pdf","markdown":"https://ijcai.org/proceedings/2020/390.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/390","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":390,"pages":{"first":2816,"last":2823},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25298/file_25298.m3u8","length_sec":292,"provider":"hls"}],"references":[{"position":1,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Pe- ter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In NIPS, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Pe-Ter Welinder, Bob Mcgrew, Josh Tobin, Openai, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. JAIR, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burda et al., 2019] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random net- work distillation. In ICLR, 2019.","title":"Exploration by random network distillation","authors":"Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Castro et al., 2018] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018.","title":"Dopamine: A Research Framework for Deep Reinforcement Learning","authors":"Castro","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Colas et al., 2018] Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms. arXiv preprint arXiv:1802.05054, 2018.","title":"Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms","authors":"Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fox et al., 2018] Lior Fox, Leshem Choshen, and Yonatan Loewenstein. {DORA} The Explorer: Directed Outreach- ing Reinforcement Action-Selection. In ICLR, 2018.","title":"{DORA} The Explorer: Directed Outreaching Reinforcement Action-Selection","authors":"Lior Fox, Leshem Choshen, Yonatan Loewenstein","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fu et al., 2017] Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. In NIPS, 2017.","title":"Ex2: Exploration with exemplar models for deep reinforcement learning","authors":"Justin Fu, John Co-Reyes, Sergey Levine","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gehring and Precup, 2013] Clement Gehring and Doina Precup. Smart Exploration in Reinforcement Learning using Absolute Temporal Difference Errors. In AAMAS, 2013.","title":"Smart Exploration in Reinforcement Learning using Absolute Temporal Difference Errors","authors":"Precup Gehring, Clement Gehring, Doina Precup","year":"2013","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hong et al., 2019] Zhang-Wei Hong, Tsu-Jui Fu, Tzu-Yun Shann, Yi-Hsiang Chang, and Chun-Yi Lee. Adversarial exploration strategy for self-supervised imitation learning. In CoRL, 2019.","title":"Adversarial exploration strategy for self-supervised imitation learning","authors":"Zhang-Wei Hong, Tsu-Jui Fu, Tzu-Yun Shann, Yi-Hsiang Chang, Chun-Yi Lee","year":"2019","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In NIPS, 2016.","title":"Vime: Variational information maximizing exploration","authors":"Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Islam et al., 2017] Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of Bench- marked Deep Reinforcement Learning Tasks for Continu- ous Control. CoRR, abs/1708.0, 2017.","title":"Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control","authors":"Islam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kalashnikov et al., 2018] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vin- cent Vanhoucke, et al. Scalable deep reinforcement learn- ing for vision-based robotic manipulation. In CoRL, 2018.","title":"Scalable deep reinforcement learning for vision-based robotic manipulation","authors":"Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke","year":"2018","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2019] Hyoungseok Kim, Jaekyeom Kim, Yeon- woo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Ex- ploration with mutual information. In ICML, 2019.","title":"Emi: Exploration with mutual information","authors":"Kim","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning: Deep Deterministic Policy Gradients (DDPG). ICLR, 2015.","title":"Continuous control with deep reinforcement learning: Deep Deterministic Policy Gradients (DDPG)","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-33, feb 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Osband et al., 2018] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforce- ment learning. In NIPS, 2018.","title":"Randomized prior functions for deep reinforcement learning","authors":"Ian Osband, John Aslanides, Albin Cassirer","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schmidhuber, 1991] Jürgen Schmidhuber. Adaptive con- fidence and adaptive curiosity. In Institut fur Infor- matik, Technische Universitat Munchen, Arcisstr. 21, 800 Munchen 2. Citeseer, 1991.","title":"Adaptive confidence and adaptive curiosity","authors":"Jürgen Schmidhuber","year":"1991","venue":"Institut fur Informatik","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simmons-Edler et al., 2019] Riley Simmons-Edler, Ben Eisner, Eric Mitchell, Sebastian Seung, and Daniel Lee. Q- learning for continuous actions with cross-entropy guided policies. arXiv preprint arXiv:1903.10605, 2019.","title":"Qlearning for continuous actions with cross-entropy guided policies","authors":"Riley Simmons-Edler, Ben Eisner, Eric Mitchell, Sebastian Seung, Daniel Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stadie et al., 2015] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing Exploration In Reinforce- ment Learning With Deep Predictive Models. CoRR, abs/1507.0, 2015.","title":"Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models","authors":"Sergey Bradly C Stadie, Pieter Levine, Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. {IEEE} Trans. Neural Networks, 9(5):1054, 1998.","title":"Reinforcement Learning: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":"{IEEE} Trans. Neural Networks","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Taiga et al., 2020] Adrien Ali Taiga, William Fedus, Mar- los C. Machado, Aaron Courville, and Marc G. Bellemare. On bonus based exploration methods in the arcade learn- ing environment. In ICLR, 2020.","title":"On bonus based exploration methods in the arcade learning environment","authors":"Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, Marc G Bellemare","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learn- ing. In NIPS, 2017.","title":"# exploration: A study of count-based exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip Deturck, Pieter Abbeel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Thrun and Möller, 1992] Sebastian B Thrun and Knut Möller. Active exploration in dynamic environments. In NIPS, 1992.","title":"Active exploration in dynamic environments","authors":"Möller ; Thrun, B Sebastian, Knut Thrun, Möller","year":"1992","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In ICML, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, Nando Freitas","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}