{"type":"ScholarlyArticle","title":"A Strongly Asymptotically Optimal Agent in General Environments","authors":[{"name":"Michael K. Cohen","slug":"michael-k-cohen","given":"Michael K.","family":"Cohen","affiliation":"Australian National University"},{"name":"Elliot Catt","slug":"elliot-catt","given":"Elliot","family":"Catt","affiliation":"Australian National University"},{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"Australian National University"}],"abstract":"Reinforcement Learning agents are expected to eventually perform well. Typically, this takes the form of a guarantee about the asymptotic behavior of an algorithm given some assumptions about the environment. We present an algorithm for a policy whose value approaches the optimal value with probability 1 in all computable probabilistic environments, provided the agent has a bounded horizon. This is known as strong asymptotic optimality, and it was previously unknown whether it was possible for a policy to be strongly asymptotically optimal in the class of all computable probabilistic environments. Our agent, Inquisitive Reinforcement Learner (Inq), is more likely to explore the more it expects an exploratory action to reduce its uncertainty about which environment it is in, hence the term inquisitive. Exploring inquisitively is a strategy that can be applied generally; for more manageable environment classes, inquisitiveness is tractable. We conducted experiments in \"grid-worlds\" to compare the Inquisitive Reinforcement Learner to other weakly asymptotically optimal agents.","doi":"10.24963/ijcai.2019/302","url":"https://ijcai.org/proceedings/2019/302","pdf":"https://ijcai.org/proceedings/2019/0302.pdf","markdown":"https://ijcai.org/proceedings/2019/302.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/302","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":302,"pages":{"first":2179,"last":2186},"keywords":["Uncertainty in AI: Exact Probabilistic Inference","Uncertainty in AI: Sequential Decision Making","Machine Learning: Reinforcement Learning","Planning and Scheduling: Model-Based Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Aslanides et al., 2017] John Aslanides, Jan Leike, and Mar- cus Hutter. Universal reinforcement learning algorithms: survey and experiments. In Proceedings of the 26th Inter- national Joint Conference on Artificial Intelligence, pages 1403-1410. AAAI Press, 2017.","title":"John Aslanides, Jan Leike, and Marcus Hutter. Universal reinforcement learning algorithms: survey and experiments","authors":"Aslanides","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aslanides, 2017] John Aslanides. AIXIjs: A software demo for general reinforcement learning. arXiv preprint arXiv:1705.07615, 2017.","title":"AIXIjs: A software demo for general reinforcement learning","authors":"John Aslanides","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hessel et al., 2018] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Sil- ver. Rainbow: Combining improvements in deep rein- forcement learning. In Proc. of AAAI Conference on Arti- ficial Intelligence, 2018.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Tom Hado Van Hasselt, Georg Schaul, Will Ostrovski, Dan Dabney, Bilal Horgan, Mohammad Piot, David Azar, Silver","year":"2018","venue":"Proc. of AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hester and Stone, 2012] Todd Hester and Peter Stone. In- trinsically motivated model learning for a developing curi- ous agent. In 2012 IEEE international conference on de- velopment and learning and epigenetic robotics (ICDL), pages 1-6. IEEE, 2012.","title":"Intrinsically motivated model learning for a developing curious agent","authors":"Stone Hester, Todd Hester, Peter Stone","year":"2012","venue":"IEEE international conference on development and learning and epigenetic robotics (ICDL)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Curiosity-driven exploration in deep reinforcement learn- ing via bayesian neural networks. arXiv preprint arXiv:1605.09674, 2016.","title":"Curiosity-driven exploration in deep reinforcement learning via bayesian neural networks","authors":"Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hutter, 2005] Marcus Hutter. Universal Artificial Intelli- gence: Sequential Decisions based on Algorithmic Proba- bility. Springer, Berlin, 2005.","title":"Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability","authors":"Marcus Hutter, Hutter","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lattimore and Hutter, 2011] Tor Lattimore and Marcus Hutter. Asymptotically optimal agents. In Proc. 22nd International Conf. on Algorithmic Learning Theory (ALT'11), volume 6925 of LNAI, pages 368-382, Espoo, Finland, 2011. Springer.","title":"Tor Lattimore and Marcus Hutter. Asymptotically optimal agents","authors":"Hutter Lattimore","year":"2011","venue":"Proc. 22nd International Conf. on Algorithmic Learning Theory (ALT'11)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lattimore and Hutter, 2014a] Tor Lattimore and Marcus Hutter. Bayesian reinforcement learning with exploration. In International Conference on Algorithmic Learning The- ory, pages 170-184. Springer, 2014.","title":"Tor Lattimore and Marcus Hutter. Bayesian reinforcement learning with exploration","authors":"Hutter Lattimore","year":"2014","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lattimore and Hutter, 2014b] Tor Lattimore and Marcus Hutter. General time consistent discounting. Theoretical Computer Science, 519:140-154, 2014.","title":"Tor Lattimore and Marcus Hutter. General time consistent discounting","authors":"Hutter Lattimore","year":"2014","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Leike et al., 2016] Jan Leike, Tor Lattimore, Laurent Orseau, and Marcus Hutter. Thompson sampling is asymp- totically optimal in general environments. In Proc. 32nd International Conf. on Uncertainty in Artificial Intelli- gence (UAI'16), pages 417-426, New Jersey, USA, 2016. AUAI Press.","title":"Thompson sampling is asymptotically optimal in general environments","authors":"Leike","year":"2016","venue":"Proc. 32nd International Conf. on Uncertainty in Artificial Intelligence (UAI'16)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Orseau et al., 2013] Laurent Orseau, Tor Lattimore, and Marcus Hutter. Universal knowledge-seeking agents for stochastic environments. In Proc. 24th International Conf. on Algorithmic Learning Theory (ALT'13), volume 8139 of LNAI, pages 158-172, Singapore, 2013. Springer.","title":"Universal knowledge-seeking agents for stochastic environments","authors":"Laurent Orseau, Tor Lattimore, Marcus Hutter","year":"2013","venue":"Proc. 24th International Conf. on Algorithmic Learning Theory (ALT'13)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-carlo planning in large pomdps. In Advances in neural information processing systems, pages 2164-2172, 2010.","title":"Monte-carlo planning in large pomdps","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Silver et al., 2017] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.","title":"Mastering chess and shogi by self-play with a general reinforcement learning algorithm","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Still, 2009] Susanne Still. Information-theoretic approach to interactive learning. EPL (Europhysics Letters), 85(2):28005, 2009.","title":"Information-theoretic approach to interactive learning","authors":"Susanne Still","year":"2009","venue":"Europhysics Letters)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Whittle, 1979] Peter Whittle. Discussion of Dr Gittins' pa- per. Journal of the Royal Statistical Society, 41:164-177, 1979.","title":"Discussion of Dr Gittins' paper","authors":"Peter Whittle, Whittle","year":"1979","venue":"Journal of the Royal Statistical Society","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}