{"type":"ScholarlyArticle","title":"On Thompson Sampling and Asymptotic Optimality","authors":[{"name":"Jan Leike","slug":"jan-leike","given":"Jan","family":"Leike","affiliation":"‡ FHI , University of Oxford"},{"name":"Tor Lattimore","slug":"tor-lattimore","given":"Tor","family":"Lattimore","affiliation":""},{"name":"Laurent Orseau","slug":"laurent-orseau","given":"Laurent","family":"Orseau","affiliation":""},{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"§ Australian National University"}],"abstract":"We discuss some recent results on Thompson sampling for nonparametric reinforcement learning in countable classes of general stochastic environments. These environments can be non-Markovian, non-ergodic, and partially observable. We show that Thompson sampling learns the environment class in the sense that\r\n(1) asymptotically its value converges in mean to the optimal value and\r\n(2) given a recoverability assumption regret is sublinear.\r\nWe conclude with a discussion about optimality in reinforcement learning.","doi":"10.24963/ijcai.2017/688","url":"https://ijcai.org/proceedings/2017/688","pdf":"https://ijcai.org/proceedings/2017/0688.pdf","markdown":"https://ijcai.org/proceedings/2017/688.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/688","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":688,"pages":{"first":4889,"last":4893},"keywords":["Artificial Intelligence: artificial intelligence","Artificial Intelligence: uncertainty in artificial intelligence","Artificial Intelligence: machine learning"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Goyal, 2011] Shipra Agrawal and Navin Goyal. Analysis of Thompson sampling for the multi- armed bandit problem. In Conference on Learning Theory, 2011.","title":"Analysis of Thompson sampling for the multiarmed bandit problem","authors":"Shipra Agrawal, Navin Goyal","year":"2011","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aslanides et al., 2017] John Aslanides, Jan Leike, and Mar- cus Hutter. Universal reinforcement learning algorithms: Survey and experiments. In International Joint Conference on Artificial Intelligence, 2017.","title":"John Aslanides, Jan Leike, and Marcus Hutter. Universal reinforcement learning algorithms: Survey and experiments","authors":"Aslanides","year":"2017","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Asmuth et al., 2009] John Asmuth, Lihong Li, Michael L Littman, Ali Nouri, and David Wingate. A Bayesian sam- pling approach to exploration in reinforcement learning. In Uncertainty in Artificial Intelligence, pages 19-26, 2009.","title":"A Bayesian sampling approach to exploration in reinforcement learning","authors":"John Asmuth, Lihong Li, Ali Michael L Littman, David Nouri, Wingate","year":"2009","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2010] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11:1563-1600, 2010.","title":"Near-optimal regret bounds for reinforcement learning","authors":"Peter Auer, Thomas Jaksch, Ronald Ortner","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bartók et al., 2014] Gábor Bartók, Dean Foster, Dávid Pál, Alexander Rakhlin, and Csaba Szepesvári. Par- tial monitoring-classification, regret bounds, and algo- rithms. Mathematics of Operations Research, 39(4):967- 997, 2014.","title":"Partial monitoring-classification, regret bounds, and algorithms","authors":"Gábor Bartók, Dean Foster, Dávid Pál, Alexander Rakhlin, Csaba Szepesvári","year":"2014","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Process- ing Systems, pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck and Bianchi, 2012] Sébastien Bubeck and Cesa- Nicolò Bianchi. Regret analysis of stochastic and non- stochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5(1):1-122, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Bianchi Bubeck, Sébastien Bubeck, Cesa-Nicolò Bianchi","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chapelle and Li, 2011] Olivier Chapelle and Lihong Li. An empirical evaluation of Thompson sampling. In Advances in Neural Information Processing Systems, pages 2249- 2257, 2011.","title":"An empirical evaluation of Thompson sampling","authors":"Li ; Chapelle, Olivier Chapelle, Lihong Li","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dearden et al., 1998] Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian Q-learning. In AAAI, pages 761- 768, 1998.","title":"Bayesian Q-learning","authors":"Richard Dearden, Nir Friedman, Stuart Russell","year":"1998","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gopalan and Mannor, 2015] Aditya Gopalan and Shie Man- nor. Thompson sampling for learning parameterized Markov decision processes. In Conference on Learning Theory, pages 861-898, 2015.","title":"Aditya Gopalan and Shie Mannor. Thompson sampling for learning parameterized Markov decision processes","authors":"Mannor Gopalan","year":"2015","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hutter, 2000] Marcus Hutter. A theory of universal artifi- cial intelligence based on algorithmic complexity. arXiv preprint cs/0004001, 2000.","title":"A theory of universal artificial intelligence based on algorithmic complexity","authors":"Marcus Hutter","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hutter, 2002] Marcus Hutter. Self-optimizing and Pareto- optimal policies in general environments based on Bayes- mixtures. In Computational Learning Theory, pages 364- 379. Springer, 2002.","title":"Self-optimizing and Paretooptimal policies in general environments based on Bayesmixtures","authors":"Marcus Hutter","year":"2002","venue":"Computational Learning Theory","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hutter, 2005] Marcus Hutter. Universal Artificial Intelli- gence. Springer, 2005.","title":"Universal Artificial Intelligence","authors":"Marcus Hutter","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaufmann et al., 2012] Emilie Kaufmann, Nathaniel Ko- rda, and Rémi Munos. Thompson sampling: An asymptot- ically optimal finite-time analysis. In Algorithmic Learn- ing Theory, pages 199-213. Springer, 2012.","title":"Thompson sampling: An asymptotically optimal finite-time analysis","authors":"Emilie Kaufmann, Nathaniel Korda, Rémi Munos","year":"2012","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lamont et al., 2017] Sean Lamont, John Aslanides, Jan Leike, and Marcus Hutter. Generalised discount func- tions applied to a Monte-Carlo AIµ implementation. In Autonomous Agents and Multiagent Systems, 2017.","title":"Generalised discount functions applied to a Monte-Carlo AIµ implementation","authors":"Sean Lamont, John Aslanides, Jan Leike, Marcus Hutter","year":"2017","venue":"Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lattimore and Hutter, 2011] Tor Lattimore and Marcus Hutter. Asymptotically optimal agents. In Algorithmic Learning Theory, pages 368-382. Springer, 2011.","title":"Tor Lattimore and Marcus Hutter. Asymptotically optimal agents","authors":"Hutter Lattimore","year":"2011","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lattimore and Hutter, 2014] Tor Lattimore and Marcus Hutter. General time consistent discounting. Theoretical Computer Science, 519:140-154, 2014.","title":"Tor Lattimore and Marcus Hutter. General time consistent discounting","authors":"Hutter Lattimore","year":"2014","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lattimore and Szepesvári, 2017] Tor Lattimore and Csaba Szepesvári. The end of optimism? An asymptotic analy- sis of finite-armed linear bandits. In AI & Statistics, pages 728-737, 2017.","title":"Tor Lattimore and Csaba Szepesvári. The end of optimism? An asymptotic analysis of finite-armed linear bandits","authors":"Szepesvári Lattimore","year":"2017","venue":"AI & Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lattimore, 2013] Tor Lattimore. Theory of General Rein- forcement Learning. PhD thesis, Australian National Uni- versity, 2013.","title":"Tor Lattimore. Theory of General Reinforcement Learning","authors":"Lattimore","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Leike and Hutter, 2015] Jan Leike and Marcus Hutter. Bad universal priors and notions of optimality. In Conference on Learning Theory, pages 1244-1259, 2015.","title":"Bad universal priors and notions of optimality","authors":"Hutter Leike, Jan Leike, Marcus Hutter","year":"2015","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Leike et al., 2016a] Jan Leike, Tor Lattimore, Laurent Orseau, and Marcus Hutter. Thompson sampling is asymp- totically optimal in general environments. In Uncertainty in Artificial Intelligence, pages 417-426, 2016.","title":"Thompson sampling is asymptotically optimal in general environments","authors":"Leike","year":"2016","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Leike et al., 2016b] Jan Leike, Jessica Taylor, and Benya Fallenstein. A formal solution to the grain of truth prob- lem. In Uncertainty in Artificial Intelligence, pages 427- 436, 2016.","title":"A formal solution to the grain of truth problem","authors":"Leike","year":"2016","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Leike, 2016a] Jan Leike. Exploration potential. In Euro- pean Workshop on Reinforcement Learning, 2016.","title":"Exploration potential","authors":"Jan Leike, Leike","year":"2016","venue":"European Workshop on Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Leike, 2016b] Jan Leike. Nonparametric General Rein- forcement Learning. PhD thesis, Australian National Uni- versity, 2016.","title":"Nonparametric General Reinforcement Learning","authors":"Jan Leike, Leike","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nguyen et al., 2013] Phuong Nguyen, Odalric-Ambrym Maillard, Daniil Ryabko, and Ronald Ortner. Competing with an infinite set of models in reinforcement learning. In Artificial Intelligence and Statistics, pages 463-471, 2013.","title":"Competing with an infinite set of models in reinforcement learning","authors":"Phuong Nguyen, Odalric-Ambrym Maillard, Daniil Ryabko, Ronald Ortner","year":"2013","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Orseau et al., 2013] Laurent Orseau, Tor Lattimore, and Marcus Hutter. Universal knowledge-seeking agents for stochastic environments. In Algorithmic Learning Theory, pages 158-172. Springer, 2013.","title":"Universal knowledge-seeking agents for stochastic environments","authors":"Laurent Orseau, Tor Lattimore, Marcus Hutter","year":"2013","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Orseau, 2013] Laurent Orseau. Asymptotic non-learnability of universal agents with computable horizon functions. Theoretical Computer Science, 473:149-156, 2013.","title":"Asymptotic non-learnability of universal agents with computable horizon functions","authors":"Laurent Orseau","year":"2013","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ortega and Braun, 2010] Pedro A Ortega and Daniel A Braun. A minimum relative entropy principle for learn- ing and acting. Journal of Artificial Intelligence Research, pages 475-511, 2010.","title":"A minimum relative entropy principle for learning and acting","authors":"Pedro A Ortega, Daniel A Braun","year":"2010","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ortega and Braun, 2012] Pedro Ortega and Daniel Braun. Adaptive coding of actions and observations. In NIPS Workshop on Information in Perception and Action, 2012.","title":"Adaptive coding of actions and observations","authors":"Pedro Ortega, Daniel Braun","year":"2012","venue":"NIPS Workshop on Information in Perception and Action","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Osband and Van Roy, 2014] Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the Eluder Dimension. In Advances in Neural Information Processing Systems, pages 1466-1474, 2014.","title":"Model-based reinforcement learning and the Eluder Dimension","authors":"Van Osband, Ian Roy, Benjamin Osband, Roy Van","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Osband and van Roy, 2016] Ian Osband and Benjamin van Roy. Why is posterior sampling better than optimism for reinforcement learning. In European Workshop on Rein- forcement Learning, 2016.","title":"Why is posterior sampling better than optimism for reinforcement learning","authors":"Van Osband, Ian Roy, Benjamin Osband, Van Roy","year":"2016","venue":"European Workshop on Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Osband et al., 2013] Ian Osband, Dan Russo, and Benjamin van Roy. (More) efficient reinforcement learning via pos- terior sampling. In Advances in Neural Information Pro- cessing Systems, pages 3003-3011, 2013.","title":"More) efficient reinforcement learning via posterior sampling","authors":"Ian Osband, Dan Russo, Benjamin Van Roy","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Russo and Van Roy, 2014] Dan Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems, pages 1583-1591, 2014.","title":"Learning to optimize via information-directed sampling","authors":"Van Russo, Dan Roy, Benjamin Russo, Roy Van","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Strens, 2000] Malcolm Strens. A Bayesian framework for reinforcement learning. In International Conference on Machine Learning, pages 943-950, 2000.","title":"A Bayesian framework for reinforcement learning","authors":"Malcolm Strens","year":"2000","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Sunehag and Hutter, 2015] Peter Sunehag and Marcus Hut- ter. Rationality, optimism and guarantees in general re- inforcement learning. Journal of Machine Learning Re- search, 16:1345-1390, 2015.","title":"Peter Sunehag and Marcus Hutter. Rationality, optimism and guarantees in general reinforcement learning","authors":"Hutter Sunehag","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, pages 285-294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}