{"type":"ScholarlyArticle","title":"Universal Reinforcement Learning Algorithms: Survey and Experiments","authors":[{"name":"John Aslanides","slug":"john-aslanides","given":"John","family":"Aslanides","affiliation":"University of Oxford"},{"name":"Jan Leike","slug":"jan-leike","given":"Jan","family":"Leike","affiliation":"‡ Australian National University Future of Humanity Institute ,"},{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"University of Oxford"}],"abstract":"Many state-of-the-art reinforcement learning (RL) algorithms typically assume that the environment is an ergodic Markov Decision Process (MDP). In contrast, the field of universal reinforcement learning (URL) is concerned with algorithms that make as few assumptions as possible about the environment. The universal Bayesian agent AIXI and a family of related URL algorithms have been developed in this setting. While numerous theoretical optimality results have been proven for these agents, there has been no empirical investigation of their behavior to date. We present a short and accessible survey of these URL algorithms under a unified notation and framework, along with results of some experiments that qualitatively illustrate some properties of the resulting policies, and their relative performance on partially-observable gridworld environments. We also present an open- source reference implementation of the algorithms which we hope will facilitate further understanding of, and experimentation with, these ideas.","doi":"10.24963/ijcai.2017/194","url":"https://ijcai.org/proceedings/2017/194","pdf":"https://ijcai.org/proceedings/2017/0194.pdf","markdown":"https://ijcai.org/proceedings/2017/194.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/194","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":194,"pages":{"first":1403,"last":1410},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Agent Theories and Models","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2016] Marc G. Bellemare, Sriram Srini- vasan, Georg Ostrovski, et al. Unifying count-based ex- ploration and intrinsic motivation. CoRR, abs/1606.01868, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"G Marc, Sriram Bellemare, Georg Srinivasan, Ostrovski","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, et al. VIME: Variational information maximizing exploration. In Advances in Neural Information Process- ing Systems 29, pages 1109-1117. 2016.","title":"VIME: Variational information maximizing exploration","authors":"Rein Houthooft, Xi Chen, Yan Duan","year":"2016","venue":"Advances in Neural Information Processing Systems 29","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hutter, 2005] Marcus Hutter. Universal Artificial Intelli- gence. Springer, 2005.","title":"Universal Artificial Intelligence","authors":"Marcus Hutter","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hutter, 2009] Marcus Hutter. Open problems in universal induction & intelligence. Algorithms, 3(2):879-906, 2009.","title":"Open problems in universal induction & intelligence","authors":"Marcus Hutter, Hutter","year":"2009","venue":"Algorithms","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Eu- ropean Conference on Machine Learning, pages 282-293, 2006.","title":"Bandit based monte-carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lamont et al., 2017] Sean Lamont, John Aslanides, Jan Leike, and Marcus Hutter. Generalised discount functions applied to a Monte-Carlo AImu implementation. In Au- tonomous Agents and Multiagent Systems, pages 1589- 1591, 2017.","title":"Generalised discount functions applied to a Monte-Carlo AImu implementation","authors":"Sean Lamont, John Aslanides, Jan Leike, Marcus Hutter","year":"2017","venue":"Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lattimore, 2013] Tor Lattimore. Theory of General Rein- forcement Learning. PhD thesis, ANU, 2013.","title":"Tor Lattimore. Theory of General Reinforcement Learning","authors":"Lattimore","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Leike and Hutter, 2015a] Jan Leike and Marcus Hutter. Bad universal priors and notions of optimality. In Conference on Learning Theory, pages 1244-1259, 2015.","title":"Bad universal priors and notions of optimality","authors":"Hutter Leike, Jan Leike, Marcus Hutter","year":"2015","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Leike and Hutter, 2015b] Jan Leike and Marcus Hutter. On the computability of Solomonoff induction and knowledge-seeking. In Algorithmic Learning Theory, pages 364-378, 2015.","title":"On the computability of Solomonoff induction and knowledge-seeking","authors":"Hutter Leike, Jan Leike, Marcus Hutter","year":"2015","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Leike et al., 2016] Jan Leike, Tor Lattimore, Laurent Orseau, and Marcus Hutter. Thompson sampling is asymp- totically optimal in general environments. In Uncertainty in Artificial Intelligence, 2016.","title":"Thompson sampling is asymptotically optimal in general environments","authors":"Leike","year":"2016","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Leike, 2016] Jan Leike. Nonparametric General Reinforce- ment Learning. PhD thesis, ANU, 2016.","title":"Nonparametric General Reinforcement Learning","authors":"Jan Leike","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Martin et al., 2017] Jarryd Martin, Suraj Narayanan S, Tom Everitt, and Marcus Hutter. Count-based exploration in feature space for reinforcement learning. In Proceedings of the Twenty-Sixth International Joint Conference on Ar- tificial Intelligence, IJCAI'17. AAAI Press, 2017.","title":"Count-based exploration in feature space for reinforcement learning","authors":"Jarryd Martin, Suraj Narayanan, S, Tom Everitt, Marcus Hutter","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI'17","doi":null,"url":null,"internal_url":"/proceedings/2017/344"},{"position":14,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Playing Atari with deep reinforcement learning. Technical report, Google DeepMind, 2013.","title":"Playing Atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep re- inforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Orseau et al., 2013] Laurent Orseau, Tor Lattimore, and Marcus Hutter. Universal knowledge-seeking agents for stochastic environments. In Algorithmic Learning Theory, pages 158-172. Springer, 2013.","title":"Universal knowledge-seeking agents for stochastic environments","authors":"Laurent Orseau, Tor Lattimore, Marcus Hutter","year":"2013","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Orseau, 2010] Laurent Orseau. Optimality issues of univer- sal greedy agents with static priors. In Algorithmic Learn- ing Theory, pages 345-359. Springer, 2010.","title":"Optimality issues of universal greedy agents with static priors","authors":"Laurent Orseau","year":"2010","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Orseau, 2011] Laurent Orseau. Universal knowledge- seeking agents. In Algorithmic Learning Theory, pages 353-367. Springer, 2011.","title":"Laurent Orseau. Universal knowledgeseeking agents","authors":"Orseau","year":"2011","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pathak et al., 2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven Exploration by Self- supervised Prediction. ArXiv e-prints, May 2017.","title":"Curiosity-driven Exploration by Selfsupervised Prediction","authors":"D Pathak, P Agrawal, A A Efros, T Darrell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rissanen, 1978] Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465-471, 1978.","title":"Modeling by shortest data description","authors":"Jorma Rissanen","year":"1978","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In Advances in Neural Information Processing Systems 23, pages 2164- 2172. 2010.","title":"Monte-Carlo planning in large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, et al. Deterministic policy gradient algorithms. In Tony Jebara and Eric P. Xing, editors, International Conference on Machine Learning, pages 387-395, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Mad- dison, et al. Mastering the game of Go with deep neu- ral networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Solomonoff, 1978] Ray Solomonoff. Complexity-based in- duction systems: Comparisons and convergence theorems. IEEE Transactions on Information Theory, 24(4):422- 432, 1978.","title":"Complexity-based induction systems: Comparisons and convergence theorems","authors":"Ray Solomonoff","year":"1978","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sunehag and Hutter, 2015] Peter Sunehag and Marcus Hut- ter. Rationality, optimism and guarantees in general re- inforcement learning. Journal of Machine Learning Re- search, 16:1345-1390, 2015.","title":"Peter Sunehag and Marcus Hutter. Rationality, optimism and guarantees in general reinforcement learning","authors":"Hutter Sunehag","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton and Barto, 1998] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Barto Sutton, Richard Sutton, Andrew Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, pages 285-294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thrun, 1992] S. Thrun. The role of exploration in learn- ing control. In Handbook for Intelligent Control: Neural, Fuzzy and Adaptive Approaches. 1992.","title":"The role of exploration in learning control","authors":"S Thrun","year":"1992","venue":"Handbook for Intelligent Control: Neural, Fuzzy and Adaptive Approaches","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Veness et al., 2011] Joel Veness, Kee Siong Ng, Marcus Hutter, William Uther, and David Silver. A Monte-Carlo AIXI approximation. Journal of Artificial Intelligence Re- search, 40(1):95-142, 2011.","title":"A Monte-Carlo AIXI approximation","authors":"Joel Veness, Kee Siong Ng, Marcus Hutter, William Uther, David Silver","year":"2011","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}