{"type":"ScholarlyArticle","title":"Adaptive Thompson Sampling Stacks for Memory Bounded Open-Loop Planning","authors":[{"name":"Thomy Phan","slug":"thomy-phan","given":"Thomy","family":"Phan","affiliation":"LMU Munich"},{"name":"Thomas Gabor","slug":"thomas-gabor","given":"Thomas","family":"Gabor","affiliation":"LMU Munich"},{"name":"Robert Müller","slug":"robert-muller","given":"Robert","family":"Müller","affiliation":"LMU Munich"},{"name":"Christoph Roch","slug":"christoph-roch","given":"Christoph","family":"Roch","affiliation":"LMU Munich"},{"name":"Claudia Linnhoff-Popien","slug":"claudia-linnhoff-popien","given":"Claudia","family":"Linnhoff-Popien","affiliation":"LMU Munich"}],"abstract":"We propose Stable Yet Memory Bounded Open-Loop (SYMBOL) planning, a general memory bounded approach to partially observable open-loop planning. SYMBOL maintains an adaptive stack of Thompson Sampling bandits, whose size is bounded by the planning horizon and can be automatically adapted according to the underlying domain without any prior domain knowledge beyond a generative model. We empirically test SYMBOL in four large POMDP benchmark problems to demonstrate its effectiveness and robustness w.r.t. the choice of hyperparameters and evaluate its adaptive memory consumption. We also compare its performance with other open-loop planning algorithms and POMCP.","doi":"10.24963/ijcai.2019/778","url":"https://ijcai.org/proceedings/2019/778","pdf":"https://ijcai.org/proceedings/2019/0778.pdf","markdown":"https://ijcai.org/proceedings/2019/778.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/778","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Planning and Scheduling","track":"Main track","number":778,"pages":{"first":5607,"last":5613},"keywords":["Planning and Scheduling: Planning under Uncertainty","Planning and Scheduling: POMDPs","Planning and Scheduling: Planning Algorithms"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Goyal, 2013] Shipra Agrawal and Navin Goyal. Further Optimal Regret Bounds for Thompson Sampling. In Artificial Intelligence and Statistics, pages 99-107, 2013.","title":"Further Optimal Regret Bounds for Thompson Sampling","authors":"Shipra Agrawal, Navin Goyal","year":"2013","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-Time Analysis of the Multiarmed Bandit Problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-Time Analysis of the Multiarmed Bandit Problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2013] Aijun Bai, Feng Wu, and Xiaoping Chen. Bayesian Mixture Modelling and Inference based Thomp- son Sampling in Monte-Carlo Tree Search. In Advances in Neural Information Processing Systems, pages 1646- 1654, 2013.","title":"Bayesian Mixture Modelling and Inference based Thompson Sampling in Monte-Carlo Tree Search","authors":"Aijun Bai, Feng Wu, Xiaoping Chen","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2014] Aijun Bai, Feng Wu, Zongzhang Zhang, and Xiaoping Chen. Thompson Sampling based Monte- Carlo Planning in POMDPs. In Proceedings of the Twenty- Fourth International Conference on Automated Planning and Scheduling, pages 29-37. AAAI Press, 2014.","title":"Thompson Sampling based Monte-Carlo Planning in POMDPs","authors":"Aijun Bai, Feng Wu, Zongzhang Zhang, Xiaoping Chen","year":"2014","venue":"Proceedings of the Twenty-Fourth International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Belzner and Gabor, 2017] Lenz Belzner and Thomas Ga- bor. Stacked Thompson Bandits. In Proceedings of the 3rd International Workshop on Software Engineering for Smart Cyber-Physical Systems, pages 18-21. IEEE Press, 2017.","title":"Stacked Thompson Bandits","authors":"Lenz Belzner, Thomas Gabor","year":"2017","venue":"Proceedings of the 3rd International Workshop on Software Engineering for Smart Cyber-Physical Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bubeck and Munos, 2010] Sébastien Bubeck and Rémi Munos. Open Loop Optimistic Planning. In COLT, pages 477-489, 2010.","title":"Open Loop Optimistic Planning","authors":"Munos Bubeck, Sébastien Bubeck, Rémi Munos","year":"2010","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chapelle and Li, 2011] Olivier Chapelle and Lihong Li. An Empirical Evaluation of Thompson Sampling. In Ad- vances in neural information processing systems, pages 2249-2257, 2011.","title":"An Empirical Evaluation of Thompson Sampling","authors":"Li ; Chapelle, Olivier Chapelle, Lihong Li","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and Act- ing in Partially Observable Stochastic Domains. Artificial intelligence, 101(1):99-134, 1998.","title":"Planning and Acting in Partially Observable Stochastic Domains","authors":"Leslie Pack, Kaelbling Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaufmann et al., 2012] Emilie Kaufmann, Nathaniel Ko- rda, and Rémi Munos. Thompson Sampling: An Asymp- totically Optimal Finite-Time Analysis. In International Conference on Algorithmic Learning Theory, pages 199- 213. Springer, 2012.","title":"Thompson Sampling: An Asymptotically Optimal Finite-Time Analysis","authors":"Emilie Kaufmann, Nathaniel Korda, Rémi Munos","year":"2012","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo Planning. In ECML, volume 6, pages 282-293. Springer, 2006.","title":"Bandit based Monte-Carlo Planning","authors":"Szepesvári ; Kocsis, Csaba Kocsis, Szepesvári","year":"2006","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lecarpentier et al., 2018] Erwan Lecarpentier, Guillaume Infantes, Charles Lesire, and Emmanuel Rachelson. Open Loop Execution of Tree-Search Algorithms. In Proceed- ings of the 27th International Joint Conference on Artifi- cial Intelligence, pages 2362-2368. IJCAI Organization, 7 2018.","title":"Open Loop Execution of Tree-Search Algorithms","authors":"Erwan Lecarpentier, Guillaume Infantes, Charles Lesire, Emmanuel Rachelson","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/327"},{"position":12,"raw":"[Perez Liebana et al., 2015] Diego Perez Liebana, Jens Dieskau, Martin Hunermund, Sanaz Mostaghim, and Simon Lucas. Open Loop Search for General Video Game Playing. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pages 337-344. ACM, 2015.","title":"Open Loop Search for General Video Game Playing","authors":"Perez Liebana","year":"2015","venue":"Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Phan et al., 2019] Thomy Phan, Lenz Belzner, Marie Kier- meier, Markus Friedrich, Kyrill Schmid, and Claudia Linnhoff-Popien. Memory Bounded Open-Loop Planning in Large POMDPs Using Thompson Sampling. 33th AAAI Conference on Artificial Intelligence, 2019.","title":"Memory Bounded Open-Loop Planning in Large POMDPs Using Thompson Sampling","authors":"Phan","year":"2019","venue":"33th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pineau et al., 2006] Joelle Pineau, Geoffrey Gordon, and Sebastian Thrun. Anytime Point-based Approximations for Large POMDPs. Journal of Artificial Intelligence Re- search, 27:335-380, 2006.","title":"Anytime Point-based Approximations for Large POMDPs","authors":"Joelle Pineau, Geoffrey Gordon, Sebastian Thrun","year":"2006","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Powley et al., 2017] Edward Powley, Peter Cowling, and Daniel Whitehouse. Memory Bounded Monte Carlo Tree Search. AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2017.","title":"Memory Bounded Monte Carlo Tree Search","authors":"Edward Powley, Peter Cowling, Daniel Whitehouse","year":"2017","venue":"AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo Planning in Large POMDPs. In Advances in neural information processing systems, pages 2164-2172, 2010.","title":"Monte-Carlo Planning in Large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Smith and Simmons, 2004] Trey Smith and Reid Simmons. Heuristic Search Value Iteration for POMDPs. In Proceed- ings of the 20th conference on Uncertainty in artificial in- telligence, pages 520-527. AUAI Press, 2004.","title":"Heuristic Search Value Iteration for POMDPs","authors":"Simmons Smith, Trey Smith, Reid Simmons","year":"2004","venue":"Proceedings of the 20th conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Somani et al., 2013] Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. DESPOT: Online POMDP Planning with Regularization. In Advances in neural information processing systems, pages 1772-1780, 2013.","title":"DESPOT: Online POMDP Planning with Regularization","authors":"Adhiraj Somani, Nan Ye, David Hsu, Wee Sun, Lee","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Thompson, 1933] William R Thompson. On the Likelihood that One Unknown Probability exceeds Another in View of the Evidence of Two Samples. Biometrika, 25(3/4):285- 294, 1933.","title":"On the Likelihood that One Unknown Probability exceeds Another in View of the Evidence of Two Samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Weinstein and Littman, 2013] Ari Weinstein and Michael L Littman. Open-loop Planning in Large-Scale Stochastic Domains. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, pages 1436-1442. AAAI Press, 2013.","title":"Open-loop Planning in Large-Scale Stochastic Domains","authors":"Littman Weinstein, Ari Weinstein, Michael L Littman","year":"2013","venue":"Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yu et al., 2005] C Yu, Jason Chuang, Brian Gerkey, G Gor- don, and Andrew Ng. Open-Loop Plans in Multi-Robot POMDPs. Technical report, Stanford CS Dept, 2005.","title":"Open-Loop Plans in Multi-Robot POMDPs","authors":"Jason Yu, Brian Chuang, G Gerkey, Andrew Gordon, Ng","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}