{"type":"ScholarlyArticle","title":"Count-Based Exploration in Feature Space for Reinforcement Learning","authors":[{"name":"Jarryd Martin","slug":"jarryd-martin","given":"Jarryd","family":"Martin","affiliation":"Research School of Computer Science , Australian National University , Canberra"},{"name":"Suraj Narayanan S.","slug":"suraj-narayanan-s","given":"Suraj Narayanan","family":"S.","affiliation":"Research School of Computer Science , Australian National University , Canberra"},{"name":"Tom Everitt","slug":"tom-everitt","given":"Tom","family":"Everitt","affiliation":"Research School of Computer Science , Australian National University , Canberra"},{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"Research School of Computer Science , Australian National University , Canberra"}],"abstract":"We introduce a new count-based optimistic exploration algorithm for Reinforcement Learning (RL) that is feasible in environments with high-dimensional state-action spaces. The success of RL algorithms in these domains depends crucially on generalisation from limited training experience. Function approximation techniques enable RL agents to generalise in order to estimate the value of unvisited states, but at present few methods enable generalisation regarding uncertainty. This has prevented the combination of scalable RL algorithms with efficient exploration strategies that drive the agent to reduce its uncertainty. We present a new method for computing a generalised state visit-count, which allows the agent to estimate the uncertainty associated with any state. Our \\phi-pseudocount achieves generalisation by exploiting same feature representation of the state space that is used for value function approximation. States that have less frequently observed features are deemed more uncertain. The \\phi-Exploration-Bonus algorithm rewards the agent for exploring in feature space rather than in the untransformed state space. The method is simpler and less computationally expensive than some previous proposals, and achieves near state-of-the-art results on high-dimensional RL benchmarks.","doi":"10.24963/ijcai.2017/344","url":"https://ijcai.org/proceedings/2017/344","pdf":"https://ijcai.org/proceedings/2017/0344.pdf","markdown":"https://ijcai.org/proceedings/2017/344.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/344","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":344,"pages":{"first":2471,"last":2478},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Planning under Uncertainty","Uncertainty in AI: Markov Decision Processes","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2016] Marc G. Bellemare, Sriram Srini- vasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and in- trinsic motivation. CoRR, abs/1606.01868, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"G Marc, Sriram Bellemare, Georg Srinivasan, Tom Ostrovski, David Schaul, Rémi Saxton, Munos","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Kakade, 2003] Sham Machandranath Kakade. On the Sam- ple Complexity of Reinforcement Learning. PhD thesis, University College London, 2003.","title":"On the Sample Complexity of Reinforcement Learning","authors":"Sham Machandranath, Kakade","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kolter and Ng, 2009] J Zico Kolter and Andrew Y Ng. Near-Bayesian exploration in polynomial time. Interna- tional Conference on Machine Learning, pages 513-520, 2009.","title":"Near-Bayesian exploration in polynomial time","authors":"Ng ; ] J Zico Kolter, Andrew Y Kolter, Ng","year":"2009","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lai and Robbins, 1985] Tze Leung Lai and Herbert Rob- bins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Liang et al., 2016] Yitao Liang, Marlos C Machado, Erik Talvitie, and Michael Bowling. State of the art control of Atari games using shallow reinforcement learning. In Autonomous Agents and Multi-Agent Systems, 2016.","title":"State of the art control of Atari games using shallow reinforcement learning","authors":"Liang","year":"2016","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei a Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational Conference on Machine Learning, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Nair et al., 2015] Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Massively parallel methods for deep reinforcement learning. CoRR, abs/1507.04296, 2015.","title":"Massively parallel methods for deep reinforcement learning","authors":"Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Osband et al., 2016a] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep explo- ration via bootstrapped DQN. CoRR, abs/1602.04621, 2016.","title":"Deep exploration via bootstrapped DQN","authors":"Osband","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Osband et al., 2016b] Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via random- ized value functions. International Conference on Ma- chine Learning, pages 1-26, 2016.","title":"Generalization and exploration via randomized value functions","authors":"Osband","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ostrovski et al., 2017] Georg Ostrovski, Marc G. Belle- mare, Aäron van den Oord, and Rémi Munos. Count- based exploration with neural density models. CoRR, abs/1703.01310, 2017.","title":"Countbased exploration with neural density models","authors":"Georg Ostrovski, Marc G Bellemare, Aäron Van Den Oord, Rémi Munos","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Philipp Moritz, Michael I Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Stadie et al., 2015] Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforce- ment learning with deep predictive models. CoRR, abs/1507.00814, 2015.","title":"Incentivizing exploration in reinforcement learning with deep predictive models","authors":"Bradly C Stadie, Sergey Levine, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Strehl and Littman, 2004] A. L. Strehl and M. L. Littman. An empirical evaluation of interval estimation for Markov decision processes. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 128-135, 2004.","title":"An empirical evaluation of interval estimation for Markov decision processes","authors":"A L Strehl, M L Littman","year":"2004","venue":"16th IEEE International Conference on Tools with Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Strehl and Littman, 2008] Alexander L Strehl and Michael L Littman. An analysis of model-based in- terval estimation for Markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331, 2008.","title":"An analysis of model-based interval estimation for Markov decision processes","authors":"L Alexander, Strehl, Michael L Littman","year":"2008","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Strehl et al., 2009] Alexander L Strehl, Lihong Li, and Michael L Littman. Reinforcement Learning in Finite MDPs : PAC Analysis. Journal of Machine Learning Re- search, 10:2413-2444, 2009.","title":"Reinforcement Learning in Finite MDPs : PAC Analysis","authors":"Lihong Alexander L Strehl, Li, Michael L Littman","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 1998] R.S. Sutton and A.G. Barto. Rein- forcement Learning: An Introduction. IEEE Transactions on Neural Networks, 9(5):1054-1054, 1998.","title":"Reinforcement Learning: An Introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton, 1990] Richard S. Sutton. Integrated architecture for learning, planning, and reacting based on approximating dynamic programming. In International Conference on Machine Learning, pages 216-224, 1990.","title":"Integrated architecture for learning, planning, and reacting based on approximating dynamic programming","authors":"Richard S Sutton","year":"1990","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang et al., 2016] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #Exploration: A study of count-based exploration for deep reinforcement learn- ing. CoRR, abs/1611.04717, 2016.","title":"#Exploration: A study of count-based exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[van Hasselt et al., 2016a] Hado van Hasselt, Arthur Guez, Matteo Hessel, and David Silver. Learning values across many orders of magnitude. CoRR, abs/1602.07714, 2016.","title":"Learning values across many orders of magnitude","authors":"Van Hasselt","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van Hasselt et al., 2016b] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble Q-learning. In AAAI, 2016.","title":"Deep reinforcement learning with double Q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Veness et al., 2012] Joel Veness, Kee Siong Ng, Marcus Hutter, and Michael Bowling. Context tree switching. In IEEE Data Compression Conference, pages 327-336, 2012.","title":"Context tree switching","authors":"Joel Veness, Kee Siong Ng, Marcus Hutter, Michael Bowling","year":"2012","venue":"IEEE Data Compression Conference","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2016] Ziyu Wang, Nando de Freitas, Tom Schaul, Matteo Hessel, Hado van Hasselt, and Marc Lanc- tot. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learn- ing, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Nando De Freitas, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}