{"type":"ScholarlyArticle","title":"ConstrainedZero: Chance-Constrained POMDP Planning Using Learned Probabilistic Failure Surrogates and Adaptive Safety Constraints","authors":[{"name":"Robert J. Moss","slug":"robert-j-moss","given":"Robert J.","family":"Moss","affiliation":"Stanford University , Stanford , CA"},{"name":"Arec Jamgochian","slug":"arec-jamgochian","given":"Arec","family":"Jamgochian","affiliation":"Stanford University , Stanford , CA"},{"name":"Johannes Fischer","slug":"johannes-fischer","given":"Johannes","family":"Fischer","affiliation":"Stanford University , Stanford , CA; Karlsruhe Institute of Technology (KIT) , Karlsruhe , Germany"},{"name":"Anthony Corso","slug":"anthony-corso","given":"Anthony","family":"Corso","affiliation":"Stanford University , Stanford , CA"},{"name":"Mykel J. Kochenderfer","slug":"mykel-j-kochenderfer","given":"Mykel J.","family":"Kochenderfer","affiliation":"Stanford University , Stanford , CA"}],"abstract":"To plan safely in uncertain environments, agents must balance utility with safety constraints. Safe planning problems can be modeled as a chance-constrained partially observable Markov decision process (CC-POMDP) and solutions often use expensive rollouts or heuristics to estimate the optimal value and action-selection policy. This work introduces the ConstrainedZero policy iteration algorithm that solves CC-POMDPs in belief space by learning neural network approximations of the optimal value and policy with an additional network head that estimates the failure probability given a belief. This failure probability guides safe action selection during online Monte Carlo tree search (MCTS). To avoid overemphasizing search based on the failure estimates, we introduce Δ-MCTS, which uses adaptive conformal inference to update the failure threshold during planning. The approach is tested on a safety-critical POMDP benchmark, an aircraft collision avoidance system, and the sustainability problem of safe CO₂ storage. Results show that by separating safety constraints from the objective we can achieve a target level of safety without optimizing the balance between rewards and costs.","doi":"10.24963/ijcai.2024/746","url":"https://ijcai.org/proceedings/2024/746","pdf":"https://ijcai.org/proceedings/2024/0746.pdf","markdown":"https://ijcai.org/proceedings/2024/746.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/746","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Planning and Scheduling","track":"Main Track","number":746,"pages":{"first":6752,"last":6760},"keywords":["Planning and Scheduling: PS: Planning under uncertainty","Planning and Scheduling: PS: POMDPs","Machine Learning: ML: Partially observable reinforcement learning and POMDPs"],"videos":[],"references":[{"position":1,"raw":"[Ayton and Williams, 2018] Benjamin J. Ayton and Brian C. Williams. Vulcan: A Monte Carlo Algorithm for Large Chance Constrained MDPs with Risk Bounding Func- tions. arXiv:1809.01220, 2018.","title":"Vulcan: A Monte Carlo Algorithm for Large Chance Constrained MDPs with Risk Bounding Functions","authors":"Benjamin J Ayton, Brian C Williams","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brázdil et al., 2020] Tomáš Brázdil, Krishnendu Chatterjee, Petr Novotnỳ, and Jiří Vahala. Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes. In AAAI Conference on Artificial Intelligence, volume 34, pages 9794-9801, 2020.","title":"Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes","authors":"Brázdil","year":"2020","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Browne et al., 2012] Cameron B. Browne, Edward Powley, et al. A Survey of Monte Carlo Tree Search Methods. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):1-43, 2012.","title":"A Survey of Monte Carlo Tree Search Methods","authors":"Cameron B Browne, Edward Powley","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Busch, 1985] Allen C. Busch. Methodology for Establish- ing A Target Level of Safety. Federal Aviation Administra- ton Technical Center, 1985.","title":"Methodology for Establishing A Target Level of Safety","authors":"Allen C Busch","year":"1985","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carpin and Thayer, 2022] Stefano Carpin and Thomas C. Thayer. Solving Stochastic Orienteering Problems with Chance Constraints Using Monte Carlo Tree Search. In International Conference on Automation Science and En- gineering (CASE), pages 1170-1177. IEEE, 2022.","title":"Solving Stochastic Orienteering Problems with Chance Constraints Using Monte Carlo Tree Search","authors":"Stefano Carpin, Thomas C Thayer","year":"2022","venue":"International Conference on Automation Science and Engineering (CASE)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Corso et al., 2022] Anthony Corso, Yizheng Wang, Markus Zechner, Jef Caers, and Mykel J. Kochenderfer. A POMDP Model for Safe Geological Carbon Sequestration. NeurIPS Workshop on Tackling Climate Change with Ma- chine Learning, 2022.","title":"A POMDP Model for Safe Geological Carbon Sequestration","authors":"Anthony Corso, Yizheng Wang, Markus Zechner, Jef Caers, Mykel J Kochenderfer","year":"2022","venue":"NeurIPS Workshop on Tackling Climate Change with Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Couëtoux et al., 2011] Adrien Couëtoux, Jean-Baptiste Hoock, et al. Continuous Upper Confidence Trees. In Learning and Intelligent Optimization. Springer, 2011.","title":"Continuous Upper Confidence Trees","authors":"Adrien Couëtoux, Jean-Baptiste Hoock","year":"2011","venue":"Learning and Intelligent Optimization","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Coulom, 2007] Rémi Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. In Com- puters and Games, 2007.","title":"Rémi Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search","authors":"Coulom","year":"2007","venue":"Computers and Games","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fischer and Tas, 2020] Johannes Fischer and Ömer Sahin Tas. Information Particle Filter Tree: An Online Algo- rithm for POMDPs with Belief-Based Rewards on Contin- uous Domains. In International Conference on Machine Learning, pages 3177-3187. PMLR, 2020.","title":"Information Particle Filter Tree: An Online Algorithm for POMDPs with Belief-Based Rewards on Continuous Domains","authors":"Tas Fischer, Johannes Fischer, Ömer Sahin Tas","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fischer et al., 2022] Johannes Fischer, Etienne Bührle, Da- nial Kamran, and Christoph Stiller. Guiding Belief Space Planning with Learned Models for Interactive Merging. In International Conference on Intelligent Transportation Systems (ITSC), pages 2542-2549, 2022.","title":"Guiding Belief Space Planning with Learned Models for Interactive Merging","authors":"Fischer","year":"2022","venue":"International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Flannery, 1989] Mark J. Flannery. Capital Regulation and Insured Banks Choice of Individual Loan Default Risks. Journal of Monetary Economics, 24(2):235-258, 1989.","title":"Capital Regulation and Insured Banks Choice of Individual Loan Default Risks","authors":"J Mark, Flannery","year":"1989","venue":"Journal of Monetary Economics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gibbs and Candes, 2021] Isaac Gibbs and Emmanuel Can- des. Adaptive Conformal Inference Under Distribution Shift. Advances in Neural Information Processing Systems (NeurIPS), 34:1660-1672, 2021.","title":"Isaac Gibbs and Emmanuel Candes. Adaptive Conformal Inference Under Distribution Shift","authors":"Candes Gibbs","year":"2021","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2018] Xin Huang, Ashkan Jasour, et al. Hy- brid Risk-Aware Conditional Planning with Applications in Autonomous Vehicles. In IEEE Conference on Deci- sion and Control (CDC), pages 3608-3614. IEEE, 2018.","title":"Hybrid Risk-Aware Conditional Planning with Applications in Autonomous Vehicles","authors":"Huang","year":"2018","venue":"IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Isom et al., 2008] Joshua D. Isom, Sean P. Meyn, and Richard D. Braatz. Piecewise Linear Dynamic Program- ming for Constrained POMDPs. In AAAI Conference on Artificial Intelligence, volume 1, pages 291-296, 2008.","title":"Piecewise Linear Dynamic Programming for Constrained POMDPs","authors":"Joshua D Isom, Sean P Meyn, Richard D Braatz","year":"2008","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jamgochian et al., 2023] Arec Jamgochian, Anthony Corso, and Mykel J. Kochenderfer. Online Planning for Con- strained POMDPs with Continuous Spaces through Dual Ascent. International Conference on Automated Planning and Scheduling (ICAPS), 33(1):198-202, 2023.","title":"Online Planning for Constrained POMDPs with Continuous Spaces through Dual Ascent","authors":"Arec Jamgochian, Anthony Corso, Mykel J Kochenderfer","year":"2023","venue":"International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kochenderfer et al., 2012] Mykel J. Kochenderfer, Jes- sica E. Holland, and James P. Chryssanthacopoulos. Next- Generation Airborne Collision Avoidance System. Lincoln Laboratory Journal, 19(1), 2012.","title":"Next-Generation Airborne Collision Avoidance System","authors":"J Mykel, Jessica E Kochenderfer, James P Holland, Chryssanthacopoulos","year":"2012","venue":"Lincoln Laboratory Journal","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kochenderfer et al., 2022] Mykel J. Kochenderfer, Tim A. Wheeler, and Kyle H. Wray. Algorithms for Decision Mak- ing. MIT Press, 2022.","title":"Algorithms for Decision Making","authors":"J Mykel, Tim A Kochenderfer, Kyle H Wheeler, Wray","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit Based Monte-Carlo Planning. In Eu- ropean Conference on Machine Learning. Springer, 2006.","title":"Bandit Based Monte-Carlo Planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lauri et al., 2022] Mikko Lauri, David Hsu, and Joni Pa- jarinen. Partially Observable Markov Decision Processes in Robotics: A Survey. IEEE Transactions on Robotics, 39(1):21-40, 2022.","title":"Partially Observable Markov Decision Processes in Robotics: A Survey","authors":"Mikko Lauri, David Hsu, Joni Pajarinen","year":"2022","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lee et al., 2018] Jongmin Lee, Geon-Hyeong Kim, Pascal Poupart, and Kee-Eung Kim. Monte-Carlo Tree Search for Constrained POMDPs. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.","title":"Monte-Carlo Tree Search for Constrained POMDPs","authors":"Jongmin Lee, Geon-Hyeong Kim, Pascal Poupart, Kee-Eung Kim","year":"2018","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Moerland et al., 2018] Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. A0C: Alpha Zero in Continuous Action Space. In ICML Planning and Learning (PAL) Workshop, 2018.","title":"A0C: Alpha Zero in Continuous Action Space","authors":"M Thomas, Joost Moerland, Aske Broekens, Catholijn M Plaat, Jonker","year":"2018","venue":"ICML Planning and Learning (PAL) Workshop","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Moss et al., 2024] Robert J. Moss, Anthony Corso, Jef Caers, and Mykel J. Kochenderfer. BetaZero: Belief- State Planning for Long-Horizon POMDPs using Learned Approximations. Reinforcement Learning Journal (RLJ), 1(1), 2024.","title":"BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations","authors":"Robert J Moss, Anthony Corso, Jef Caers, Mykel J Kochenderfer","year":"2024","venue":"Reinforcement Learning Journal","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ong et al., 2010] Sylvie C. W. Ong, Shao Wei Png, David Hsu, and Wee Sun Lee. Planning under Uncertainty for Robotic Tasks with Mixed Observability. The Interna- tional Journal of Robotics Research, 29(8), 2010.","title":"Planning under Uncertainty for Robotic Tasks with Mixed Observability","authors":"C W Sylvie, Shao Ong, David Wei Png, Wee Hsu, Lee Sun","year":"2010","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pajarinen and Kyrki, 2017] Joni Pajarinen and Ville Kyrki. Robotic Manipulation of Multiple Objects as a POMDP. Artificial Intelligence, 247:213-228, 2017.","title":"Joni Pajarinen and Ville Kyrki. Robotic Manipulation of Multiple Objects as a POMDP","authors":"Kyrki Pajarinen","year":"2017","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pajarinen et al., 2022] Joni Pajarinen, Jens Lundell, and Ville Kyrki. POMDP Planning Under Object Composition Uncertainty: Application to Robotic Manipulation. IEEE Transactions on Robotics, 39(1):41-56, 2022.","title":"POMDP Planning Under Object Composition Uncertainty: Application to Robotic Manipulation","authors":"Pajarinen","year":"2022","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Parthasarathy et al., 2023] Dinesh Parthasarathy, Georgios Kontes, Axel Plinge, and Christopher Mutschler. C- MCTS: Safe Planning with Monte Carlo Tree Search. arXiv:2305.16209, 2023.","title":"C-MCTS: Safe Planning with Monte Carlo Tree Search","authors":"Dinesh Parthasarathy, Georgios Kontes, Axel Plinge, Christopher Mutschler","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Platt Jr. et al., 2010] Robert Platt Jr., Russ Tedrake, Leslie Kaelbling, and Tomas Lozano-Perez. Belief Space Planning Assuming Maximum Likelihood Observations. Robotics: Science and Systems VI, 2010.","title":"Belief Space Planning Assuming Maximum Likelihood Observations","authors":"Robert Platt, Leslie Russ Tedrake, Tomas Kaelbling, Lozano-Perez","year":"2010","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Santana et al., 2016] Pedro Santana, Sylvie Thiébaux, and Brian Williams. RAO*: An Algorithm for Chance- Constrained POMDPs. In AAAI Conference on Artificial Intelligence, volume 30, 2016.","title":"RAO*: An Algorithm for Chance-Constrained POMDPs","authors":"Pedro Santana, Sylvie Thiébaux, Brian Williams","year":"2016","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Schrittwieser et al., 2020] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839), 2020.","title":"Mastering Atari, Go, chess and shogi by planning with a learned model","authors":"Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver","year":"2020","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Silver et al., 2016] David Silver, Aja Huang, et al. Master- ing the game of Go with deep neural networks and tree search. Nature, 529(7587), 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang","year":null,"venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, et al. Mastering the game of Go without human knowledge. Na- ture, 550(7676), 2017.","title":"Mastering the game of Go without human knowledge","authors":"David Silver, Julian Schrittwieser","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, et al. A general reinforcement learning algorithm that mas- ters chess, shogi, and Go through self-play. Science, 362(6419), 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play","authors":"David Silver, Thomas Hubert","year":"6419","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sokota et al., 2021] Samuel Sokota, Caleb Y. Ho, et al. Monte Carlo Tree Search with Iteratively Refining State Abstractions. Advances in Neural Information Processing Systems (NeurIPS), 34:18698-18709, 2021.","title":"Monte Carlo Tree Search with Iteratively Refining State Abstractions","authors":"Sokota","year":"2021","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Soualhia et al., 2020] Mbarka Soualhia, Foutse Khomh, and Sofiène Tahar. A Dynamic and Failure-Aware Task Scheduling Framework for Hadoop. IEEE Transactions on Cloud Computing, 8(2):553-569, 2020.","title":"Mbarka Soualhia, Foutse Khomh, and Sofiène Tahar. A Dynamic and Failure-Aware Task Scheduling Framework for Hadoop","authors":"Soualhia","year":"2020","venue":"IEEE Transactions on Cloud Computing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sunberg and Kochenderfer, 2018] Zachary N. Sunberg and Mykel J. Kochenderfer. Online Algorithms for POMDPs with Continuous State, Action, and Observation Spaces. In International Conference on Automated Planning and Scheduling (ICAPS), volume 28, 2018.","title":"Online Algorithms for POMDPs with Continuous State, Action, and Observation Spaces","authors":"Zachary N Sunberg, J Mykel, Kochenderfer","year":"2018","venue":"International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Thrun et al., 2005] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics. MIT Press, 2005.","title":"Probabilistic Robotics","authors":"Sebastian Thrun, Wolfram Burgard, Dieter Fox","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wan and Van Der Merwe, 2000] Eric A. Wan and Rudolph Van Der Merwe. The Unscented Kalman Filter for Non- linear Estimation. In Adaptive Systems for Signal Pro- cessing, Communications, and Control Symposium (AS- SPCC), pages 153-158. IEEE, 2000.","title":"The Unscented Kalman Filter for Nonlinear Estimation","authors":"Wan, Van Der Merwe ;, Eric A Wan, Rudolph Van Der Merwe","year":"2000","venue":"Adaptive Systems for Signal Processing, Communications, and Control Symposium (AS-SPCC)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}