{"type":"ScholarlyArticle","title":"Sparse Tree Search Optimality Guarantees in POMDPs with Continuous Observation Spaces","authors":[{"name":"Michael H. Lim","slug":"michael-h-lim","given":"Michael H.","family":"Lim","affiliation":"University of California , Berkeley"},{"name":"Claire Tomlin","slug":"claire-tomlin","given":"Claire","family":"Tomlin","affiliation":"University of California , Berkeley"},{"name":"Zachary N. Sunberg","slug":"zachary-n-sunberg","given":"Zachary N.","family":"Sunberg","affiliation":"University of Colorado , Boulder"}],"abstract":"Partially observable Markov decision processes (POMDPs) with continuous state and observation spaces have powerful flexibility for representing real-world decision and control problems but are notoriously difficult to solve. Recent online sampling-based algorithms that use observation likelihood weighting have shown unprecedented effectiveness in domains with continuous observation spaces. However there has been no formal theoretical justification for this technique. This work offers such a justification, proving that a simplified algorithm, partially observable weighted sparse sampling (POWSS), will estimate Q-values accurately with high probability and can be made to perform arbitrarily near the optimal solution by increasing computational power.","doi":"10.24963/ijcai.2020/572","url":"https://ijcai.org/proceedings/2020/572","pdf":"https://ijcai.org/proceedings/2020/0572.pdf","markdown":"https://ijcai.org/proceedings/2020/572.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/572","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Planning and Scheduling","track":"Main track","number":572,"pages":{"first":4135,"last":4142},"keywords":["Planning and Scheduling: POMDPs","Planning and Scheduling: Real-time Planning","Planning and Scheduling: Planning under Uncertainty"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25283/file_25283.m3u8","length_sec":289,"provider":"hls"}],"references":[{"position":1,"raw":"[Agha-Mohammadi et al., 2011] Ali-Akbar Agha- Mohammadi, Suman Chakravorty, and Nancy M. Amato. FIRM: Feedback controller-based information- state roadmap -a framework for motion planning under uncertainty. In IEEE/RSJ IROS, 2011.","title":"FIRM: Feedback controller-based informationstate roadmap -a framework for motion planning under uncertainty","authors":"Agha-Mohammadi","year":"2011","venue":"IEEE/RSJ IROS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ayer et al., 2012] Turgay Ayer, Oguzhan Alagoz, and Natasha K. Stout. A POMDP approach to personalize mammography screening decisions. Operations Research, 60(5):1019-1034, 2012.","title":"A POMDP approach to personalize mammography screening decisions","authors":"Turgay Ayer, Oguzhan Alagoz, Natasha K Stout","year":"2012","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2014] Haoyu Bai, David Hsu, and Wee Sun Lee. Integrated perception and planning in the continu- ous space: A POMDP approach. International Journal of Robotics Research, 33(9):1288-1302, 2014.","title":"Integrated perception and planning in the continuous space: A POMDP approach","authors":"Haoyu Bai, David Hsu, Wee Sun, Lee","year":"2014","venue":"International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2015] Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, and Wee Sun Lee. Intention-aware online POMDP planning for autonomous driving in a crowd. In IEEE ICRA, pages 454-460, 2015.","title":"Intention-aware online POMDP planning for autonomous driving in a crowd","authors":"Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, Wee Sun, Lee","year":"2015","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bertsekas, 2005] D. Bertsekas. Dynamic Programming and Optimal Control. Athena, 2005.","title":"Dynamic Programming and Optimal Control","authors":"D Bertsekas","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bjarnason et al., 2009] Ronald Bjarnason, Alan Fern, and Prasad Tadepalli. Lower bounding Klondike solitaire with Monte-Carlo planning. In ICAPS, 2009.","title":"Lower bounding Klondike solitaire with Monte-Carlo planning","authors":"Ronald Bjarnason, Alan Fern, Prasad Tadepalli","year":"2009","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bry and Roy, 2011] Adam Bry and Nicholas Roy. Rapidly- exploring random belief trees for motion planning under uncertainty. In IEEE ICRA, pages 723-730, 2011.","title":"Rapidlyexploring random belief trees for motion planning under uncertainty","authors":"Roy ; Bry, Adam Bry, Nicholas Roy","year":"2011","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cassandra et al., 1997] Anthony Cassandra, Michael L. Littman, and Nevin L. Zhang. Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. In UAI, pages 54-61, 1997.","title":"Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes","authors":"Anthony Cassandra, Michael L Littman, Nevin L Zhang","year":"1997","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cassandra, 1998] Anthony R. Cassandra. A survey of POMDP applications. In AAAI Fall Symposium: Planning with POMDPs, 1998.","title":"A survey of POMDP applications","authors":"Anthony R Cassandra","year":"1998","venue":"AAAI Fall Symposium: Planning with POMDPs","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Couëtoux et al., 2011] Adrien Couëtoux, Jean-Baptiste Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nico- las Bonnard. Continuous upper confidence trees. In Learning and Intelligent Optimization, Rome, Italy, 2011.","title":"Continuous upper confidence trees","authors":"Adrien Couëtoux, Jean-Baptiste Hoock, Nataliya Sokolovska, Olivier Teytaud, Nicolas Bonnard","year":"2011","venue":"Learning and Intelligent Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Egorov et al., 2017] Maxim Egorov, Zachary N. Sunberg, Edward Balaban, Tim A. Wheeler, Jayesh K. Gupta, and Mykel J. Kochenderfer. POMDPs.jl: A framework for se- quential decision making under uncertainty. Journal of Machine Learning Research, 18(26):1-5, 2017.","title":"POMDPs.jl: A framework for sequential decision making under uncertainty","authors":"Maxim Egorov, Zachary N Sunberg, Edward Balaban, Tim A Wheeler, Jayesh K Gupta, Mykel J Kochenderfer","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Garg et al., 2019] Neha P. Garg, David Hsu, and Wee Sun Lee. DESPOT-α: Online POMDP planning with large state and observation spaces. In RSS, 2019.","title":"DESPOT-α: Online POMDP planning with large state and observation spaces","authors":"P Neha, David Garg, Wee Hsu, Lee Sun","year":"2019","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hoey and Poupart, 2005] Jesse Hoey and Pascal Poupart. Solving POMDPs with continuous or large discrete obser- vation spaces. In IJCAI, pages 1332-1338, 2005.","title":"Solving POMDPs with continuous or large discrete observation spaces","authors":"Poupart Hoey, Jesse Hoey, Pascal Poupart","year":"2005","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2005/172"},{"position":14,"raw":"[Holland et al., 2013] Jessica E. Holland, Mykel J. Kochen- derfer, and Wesley A. Olson. Optimizing the next gen- eration collision avoidance system for safe, suitable, and acceptable operational performance. Air Traffic Control Quarterly, 21(3):275-297, 2013.","title":"Optimizing the next generation collision avoidance system for safe, suitable, and acceptable operational performance","authors":"Jessica E Holland, J Mykel, Wesley A Kochenderfer, Olson","year":"2013","venue":"Air Traffic Control Quarterly","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and act- ing in partially observable stochastic domains. Artificial Intelligence, 101(1):99 -134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kearns et al., 2002] Michael Kearns, Yishay Mansour, and Andrew Y. Ng. A sparse sampling algorithm for near- optimal planning in large Markov decision processes. Ma- chine Learning, 49(2):193-208, 2002.","title":"A sparse sampling algorithm for nearoptimal planning in large Markov decision processes","authors":"Michael Kearns, Yishay Mansour, Andrew Y Ng","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kochenderfer, 2015] Mykel J. Kochenderfer. Decision Making Under Uncertainty: Theory and Application. MIT Press, 2015.","title":"Decision Making Under Uncertainty: Theory and Application","authors":"J Mykel, Kochenderfer","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kurniawati and Yadav, 2016] Hanna Kurniawati and Vinay Yadav. An online POMDP solver for uncertainty plan- ning in dynamic environment. In Robotics Research, pages 611-629. Springer, 2016.","title":"An online POMDP solver for uncertainty planning in dynamic environment","authors":"Yadav Kurniawati, Hanna Kurniawati, Vinay Yadav","year":"2016","venue":"Robotics Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2008] Wee Sun Lee, Nan Rong, and David Hsu. What makes some POMDP problems easy to approxi- mate? In NeuRIPS, pages 689-696, 2008.","title":"What makes some POMDP problems easy to approximate? In NeuRIPS","authors":"Wee Sun Lee, Nan Rong, David Hsu","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[McAllester and Singh, 1999] David A. McAllester and Satinder Singh. Approximate planning for factored POMDPs using belief state simplification. In UAI, pages 409-416, 1999.","title":"Approximate planning for factored POMDPs using belief state simplification","authors":"David A Mcallester, Satinder Singh","year":"1999","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Metelli et al., 2018] Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimiza- tion via importance sampling. In NeuRIPS, pages 5442- 5454, 2018.","title":"Policy optimization via importance sampling","authors":"Alberto Maria Metelli, Matteo Papini, Francesco Faccio, Marcello Restelli","year":"2018","venue":"NeuRIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos H. Papadim- itriou and John N. Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research, 12(3):441-450, 1987.","title":"The complexity of Markov decision processes","authors":"Christos H Papadimitriou, John N Tsitsiklis","year":"1987","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Platt et al., 2010] Robert Platt, Jr., Russ Tedrake, Leslie Kaelbling, and Tomas Lozano-Perez. Belief space plan- ning assuming maximum likelihood observations. In RSS, 2010.","title":"Belief space planning assuming maximum likelihood observations","authors":"Robert Platt, Jr Russ Tedrake, Leslie Kaelbling, Tomas Lozano-Perez","year":"2010","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In NeuRIPS, pages 2164-2172, 2010.","title":"Monte-Carlo planning in large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"NeuRIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Singh and Yee, 1994] Satinder P. Singh and Richard C. Yee. An upper bound on the loss from approximate optimal- value functions. Machine Learning, 16(3):227-233, 1994.","title":"An upper bound on the loss from approximate optimalvalue functions","authors":"P Satinder, Richard C Singh, Yee","year":"1994","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sunberg and Kochenderfer, 2018] Zachary Sunberg and Mykel J. Kochenderfer. Online algorithms for POMDPs with continuous state, action, and observation spaces. In ICAPS, 2018.","title":"Online algorithms for POMDPs with continuous state, action, and observation spaces","authors":"Kochenderfer ; Sunberg, Zachary Sunberg, Mykel J Kochenderfer","year":"2018","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sunberg et al., 2013] Zachary Sunberg, Suman Chakra- vorty, and R Scott Erwin. Information space reced- ing horizon control. IEEE Transactions on Cybernetics, 43(6):2255-2260, 2013.","title":"Information space receding horizon control","authors":"Zachary Sunberg, Suman Chakravorty, R Scott, Erwin","year":"2013","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sunberg et al., 2017] Zachary N. Sunberg, Christopher J. Ho, and Mykel J. Kochenderfer. The value of inferring the internal state of traffic participants for autonomous free- way driving. In ACC, 2017.","title":"The value of inferring the internal state of traffic participants for autonomous freeway driving","authors":"Zachary N Sunberg, Christopher J Ho, Mykel J Kochenderfer","year":"2017","venue":"ACC","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Van Den Berg et al., 2012] Jur Van Den Berg, Sachin Patil, and Ron Alterovitz. Motion planning under uncertainty using iterative local optimization in belief space. Interna- tional Journal of Robotics Research, 31(11):1263-1278, 2012.","title":"Motion planning under uncertainty using iterative local optimization in belief space","authors":"Van Den, Berg","year":"2012","venue":"International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ye et al., 2017] Nan Ye, Adhiraj Somani, David Hsu, and Wee Sun Lee. DESPOT: Online POMDP planning with regularization. Journal of Artificial Intelligence Research, 58:231-266, 2017.","title":"DESPOT: Online POMDP planning with regularization","authors":"Nan Ye, Adhiraj Somani, David Hsu, Wee Sun, Lee","year":"2017","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Young et al., 2013] Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. POMDP-based statisti- cal spoken dialog systems: A review. IEEE, 101(5):1160- 1179, 2013.","title":"POMDP-based statistical spoken dialog systems: A review","authors":"Steve Young, Milica Gašić, Blaise Thomson, Jason D Williams","year":"2013","venue":"IEEE","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}