{"type":"ScholarlyArticle","title":"A Finite-State Controller Based Offline Solver for Deterministic POMDPs","authors":[{"name":"Alex Schutz","slug":"alex-schutz","given":"Alex","family":"Schutz","affiliation":"University of Oxford"},{"name":"Yang You","slug":"yang-you","given":"Yang","family":"You","affiliation":"UK Atomic Energy Authority {alexschutz ,"},{"name":"Matías Mattamala","slug":"matias-mattamala","given":"Matías","family":"Mattamala","affiliation":"University of Oxford"},{"name":"Ipek Caliskanelli","slug":"ipek-caliskanelli","given":"Ipek","family":"Caliskanelli","affiliation":"UK Atomic Energy Authority {alexschutz ,"},{"name":"Bruno Lacerda","slug":"bruno-lacerda","given":"Bruno","family":"Lacerda","affiliation":"University of Oxford"},{"name":"Nick Hawes","slug":"nick-hawes","given":"Nick","family":"Hawes","affiliation":"University of Oxford"}],"abstract":"Deterministic partially observable Markov decision processes (DetPOMDPs) often arise in planning problems where the agent is uncertain about its environmental state but can act and observe deterministically. In this paper, we propose DetMCVI, an adaptation of the Monte Carlo Value Iteration (MCVI) algorithm for DetPOMDPs, which builds policies in the form of finite-state controllers (FSCs). DetMCVI solves large problems with a high success rate, outperforming existing baselines for DetPOMDPs. We also verify the performance of the algorithm in a real-world mobile robot forest mapping scenario.","doi":"10.24963/ijcai.2025/958","url":"https://ijcai.org/proceedings/2025/958","pdf":"https://ijcai.org/proceedings/2025/0958.pdf","markdown":"https://ijcai.org/proceedings/2025/958.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/958","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Planning and Scheduling","track":"Main Track","number":958,"pages":{"first":8617,"last":8625},"keywords":["Planning and Scheduling: PS: Robot planning","Planning and Scheduling: PS: Planning algorithms","Planning and Scheduling: PS: POMDPs","Planning and Scheduling: PS: Planning under uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Amato et al., 2010] Christopher Amato, Daniel S. Bern- stein, and Shlomo Zilberstein. Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs. Autonomous Agents and Multi-Agent Systems, 21(3):293-320, November 2010.","title":"Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2010","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andriushchenko et al., 2022] Roman Andriushchenko, Mi- lan Češka, Sebastian Junges, and Joost-Pieter Katoen. In- ductive synthesis of finite-state controllers for POMDPs. In Proceedings of the Thirty-Eighth Conference on Uncer- tainty in Artificial Intelligence, pages 85-95. PMLR, Au- gust 2022.","title":"Roman Andriushchenko, Milan Češka, Sebastian Junges, and Joost-Pieter Katoen. Inductive synthesis of finite-state controllers for POMDPs","authors":"Andriushchenko","year":"2022","venue":"Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2011] Haoyu Bai, David Hsu, Wee Sun Lee, and Vien A Ngo. Monte Carlo value iteration for continuous- state POMDPs. In Algorithmic Foundations of Robotics IX: Selected Contributions of the Ninth International Workshop on the Algorithmic Foundations of Robotics, pages 175-191. Springer, 2011.","title":"Monte Carlo value iteration for continuousstate POMDPs","authors":"Haoyu Bai, David Hsu, Wee Sun Lee, Vien A Ngo","year":"2011","venue":"Algorithmic Foundations of Robotics IX: Selected Contributions of the Ninth International Workshop on the Algorithmic Foundations of Robotics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2013] Haoyu Bai, David Hsu, and Wee Sun Lee. Planning how to learn. In 2013 IEEE International Con- ference on Robotics and Automation, pages 2853-2859, Karlsruhe, Germany, May 2013. IEEE.","title":"Planning how to learn","authors":"Haoyu Bai, David Hsu, Wee Sun, Lee","year":"2013","venue":"2013 IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barto et al., 1995] Andrew G. Barto, Steven J. Bradtke, and Satinder P. Singh. Learning to act using real-time dy- namic programming. Artificial Intelligence, 72(1-2):81- 138, January 1995.","title":"Learning to act using real-time dynamic programming","authors":"Andrew G Barto, Steven J Bradtke, Satinder P Singh","year":"1995","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bonet and Geffner, 2021] Blai Bonet and Hector Geffner. Action Selection for MDPs: Anytime AO* Versus UCT. Proceedings of the AAAI Conference on Artificial Intelli- gence, 26(1):1749-1755, September 2021.","title":"Action Selection for MDPs: Anytime AO* Versus UCT","authors":"Geffner Bonet, Blai Bonet, Hector Geffner","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bonet, 2009] Blai Bonet. Deterministic POMDPs Revis- ited. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI), pages 59-66, Montreal, Canada, 2009. AUAI Press.","title":"Deterministic POMDPs Revisited","authors":"Blai Bonet","year":"2009","venue":"Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bonet, 2010] Blai Bonet. Conformant plans and beyond: Principles and complexity. Artificial Intelligence, 174(3- 4):245-269, March 2010.","title":"Conformant plans and beyond: Principles and complexity","authors":"Blai Bonet, Bonet","year":"2010","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Brafman and Shani, 2021] Ronen Brafman and Guy Shani. A Multi-Path Compilation Approach to Contingent Plan- ning. Proceedings of the AAAI Conference on Artificial Intelligence, 26(1):1868-1874, September 2021.","title":"A Multi-Path Compilation Approach to Contingent Planning","authors":"Shani ; Brafman, Ronen Brafman, Guy Shani","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chakrabarti, 1994] P.P. Chakrabarti. Algorithms for search- ing explicit AND/OR graphs and their applications to problem reduction search. Artificial Intelligence, 65(2):329-345, February 1994.","title":"Algorithms for searching explicit AND/OR graphs and their applications to problem reduction search","authors":"P P Chakrabarti","year":"1994","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chatterjee et al., 2016] Krishnendu Chatterjee, Martin Chmelik, Raghav Gupta, and Ayush Kanodia. Optimal cost almost-sure reachability in POMDPs. Artificial Intelligence, 234:26-48, 2016.","title":"Optimal cost almost-sure reachability in POMDPs","authors":"Krishnendu Chatterjee, Martin Chmelik, Raghav Gupta, Ayush Kanodia","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chatterjee et al., 2020] Krishnendu Chatterjee, Martin Chmelík, Deep Karkhanis, Petr Novotný, and Amélie Royer. Multiple-Environment Markov Decision Pro- cesses: Efficient Analysis and Applications. Proceedings of the International Conference on Automated Planning and Scheduling, 30:48-56, June 2020.","title":"Multiple-Environment Markov Decision Processes: Efficient Analysis and Applications","authors":"Krishnendu Chatterjee, Martin Chmelík, Deep Karkhanis, Petr Novotný, Amélie Royer","year":"2020","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2016] Min Chen, Emilio Frazzoli, David Hsu, and Wee Sun Lee. POMDP-lite for robust robot planning under uncertainty. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 5427-5433, Stockholm, Sweden, May 2016. IEEE.","title":"POMDP-lite for robust robot planning under uncertainty","authors":"Chen","year":"2016","venue":"2016 IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Chung and Huang, 2011] Shu-Yun Chung and Han-Pang Huang. Robot Motion Planning in Dynamic Uncertain Environments. Advanced Robotics, 25(6-7):849-870, Jan- uary 2011.","title":"Robot Motion Planning in Dynamic Uncertain Environments","authors":"Chung, Huang ; ] Shu-Yun, Chung, Han-Pang Huang","year":"2011","venue":"Advanced Robotics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dey et al., 2014] Debadeepta Dey, Andrey Kolobov, Rich Caruana, Ece Kamar, Eric Horvitz, and Ashish Kapoor. Gauss meets Canadian traveler: Shortest-path problems with correlated natural dynamics. In AAMAS'14 Proceed- ings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems, pages 1101-1108, 2014.","title":"Gauss meets Canadian traveler: Shortest-path problems with correlated natural dynamics","authors":"Dey","year":"2014","venue":"AAMAS'14 Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Duff, 2002] Michael O'Gordon Duff. Optimal Learn- ing: Computational Procedures for Bayes-adaptive Markov Decision Processes. University of Massachusetts Amherst, 2002.","title":"Optimal Learning: Computational Procedures for Bayes-adaptive Markov Decision Processes","authors":"O'gordon Michael, Duff","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ermis et al., 2021] Melike Ermis, Mingyu Park, and Insoon Yang. On Anderson Acceleration for Partially Observable Markov Decision Processes. In 2021 60th IEEE Confer- ence on Decision and Control (CDC), pages 4478-4485, Austin, TX, USA, December 2021. IEEE.","title":"On Anderson Acceleration for Partially Observable Markov Decision Processes","authors":"Melike Ermis, Mingyu Park, Insoon Yang","year":"2021","venue":"60th IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Eyerich et al., 2010] Patrick Eyerich, Thomas Keller, and Malte Helmert. High-quality policies for the canadian traveler's problem. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 24, pages 51-58, 2010.","title":"High-quality policies for the canadian traveler's problem","authors":"Patrick Eyerich, Thomas Keller, Malte Helmert","year":"2010","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ferguson et al., 2004] D. Ferguson, A. Stentz, and S. Thrun. PAO for planning with hidden state. In IEEE International Conference on Robotics and Automation, 2004. Proceed- ings. ICRA '04. 2004, pages 2840-2847 Vol.3, New Or- leans, LA, USA, 2004. IEEE.","title":"PAO for planning with hidden state","authors":"D Ferguson, A Stentz, S Thrun","year":"2004","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Guo and Barfoot, 2019] Hengwei Guo and Timothy D. Bar- foot. The Robust Canadian Traveler Problem Applied to Robot Routing. In 2019 International Conference on Robotics and Automation (ICRA), pages 5523-5529, Mon- treal, QC, Canada, May 2019. IEEE.","title":"The Robust Canadian Traveler Problem Applied to Robot Routing","authors":"Barfoot Guo, Hengwei Guo, Timothy D Barfoot","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hansen and Zilberstein, 2001] Eric A Hansen and Shlomo Zilberstein. LAO*: A heuristic search algorithm that finds solutions with loops. Artificial Intelligence, 129(1-2):35- 62, 2001.","title":"LAO*: A heuristic search algorithm that finds solutions with loops","authors":"Zilberstein Hansen, Eric A Hansen, Shlomo Zilberstein","year":"2001","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Horák et al., 2018] Karel Horák, Branislav Bosanský, and Krishnendu Chatterjee. Goal-HSVI: Heuristic Search Value Iteration for Goal POMDPs. In IJCAI, pages 4764- 4770, 2018.","title":"Goal-HSVI: Heuristic Search Value Iteration for Goal POMDPs","authors":"Karel Horák, Branislav Bosanský, Krishnendu Chatterjee","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/662"},{"position":23,"raw":"[Huang et al., 2023] Yizhou Huang, Hamza Dugmag, Tim- othy D. Barfoot, and Florian Shkurti. Stochastic Plan- ning for ASV Navigation Using Satellite Images. In 2023 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 1055-1061, London, United Kingdom, May 2023. IEEE.","title":"Stochastic Planning for ASV Navigation Using Satellite Images","authors":"Yizhou Huang, Hamza Dugmag, Timothy D Barfoot, Florian Shkurti","year":"2023","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Junges et al., 2018] Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, and Bernd Becker. Finite-state Controllers of POMDPs via Parameter Synthesis. Conference on Uncer- tainty in Artificial Intelligence, 2018.","title":"Finite-state Controllers of POMDPs via Parameter Synthesis","authors":"Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, Bernd Becker","year":"2018","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kurniawati et al., 2009] Hanna Kurniawati, David Hsu, and Wee Sun Lee. SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces. 2009.","title":"SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces","authors":"Hanna Kurniawati, David Hsu, Wee Sun, Lee","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lacerda et al., 2019] Bruno Lacerda, Fatma Faruq, David Parker, and Nick Hawes. Probabilistic planning with for- mal performance guarantees for mobile service robots. The International Journal of Robotics Research, 38(9):1098- 1123, 2019.","title":"Probabilistic planning with formal performance guarantees for mobile service robots","authors":"Lacerda","year":"2019","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Littman et al., 1995] Michael L. Littman, Anthony R. Cas- sandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. In Ma- chine Learning Proceedings 1995, pages 362-370. Else- vier, 1995.","title":"Learning policies for partially observable environments: Scaling up","authors":"L Michael, Anthony R Littman, Leslie Cassandra, Kaelbling Pack","year":"1995","venue":"Machine Learning Proceedings","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Littman, 1996] Michael Lederman Littman. Algorithms for Sequential Decision-Making. Brown University, 1996.","title":"Algorithms for Sequential Decision-Making","authors":"Michael Lederman, Littman","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mausam and Kolobov, 2012] Mausam and Andrey Kolobov. Planning with Markov Decision Processes: An AI Perspective. Synthesis Lectures on Artificial Intelligence and Machine Learning. Springer International Publishing, Cham, 2012.","title":"Mausam and Andrey Kolobov","authors":"Kolobov Mausam","year":"2012","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Muise et al., 2014] Christian Muise, Vaishak Belle, and Sheila McIlraith. Computing Contingent Plans via Fully Observable Non-Deterministic Planning. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), June 2014.","title":"Computing Contingent Plans via Fully Observable Non-Deterministic Planning","authors":"Muise","year":"2014","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Nardi and Stachniss, 2020] Lorenzo Nardi and Cyrill Stach- niss. Long-Term Robot Navigation in Indoor Environ- ments Estimating Patterns in Traversability Changes. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 300-306, Paris, France, May 2020. IEEE.","title":"Long-Term Robot Navigation in Indoor Environments Estimating Patterns in Traversability Changes","authors":"Lorenzo Nardi, Cyrill Stachniss","year":"2020","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Papadimitriou and Yannakakis, 1989] Christos H Papadim- itriou and Mihalis Yannakakis. Shortest paths without a map. In Automata, Languages and Programming: 16th International Colloquium Stresa, Italy, July 11-15, 1989 Proceedings 16, pages 610-620. Springer, 1989.","title":"Shortest paths without a map","authors":"Christos H Papadimitriou, Mihalis Yannakakis","year":"1989","venue":"Automata, Languages and Programming: 16th International Colloquium","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Raskin and Sankur, 2014] Jean-Franc ¸ois Raskin and Ocan Sankur. Multiple-Environment Markov Decision Pro- cesses, December 2014.","title":"Jean-Franc ¸ois Raskin and Ocan Sankur","authors":"Sankur Raskin","year":"2014","venue":"Multiple-Environment Markov Decision Processes","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Russell and Norvig, 2021] S. Russell and P. Norvig. Arti- ficial Intelligence: A Modern Approach, Global Edition. Pearson Education, 2021.","title":"Artificial Intelligence: A Modern Approach","authors":"S Russell, P Norvig","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shani et al., 2013] Guy Shani, Joelle Pineau, and Robert Kaplow. A survey of point-based POMDP solvers. Au- tonomous Agents and Multi-Agent Systems, 27(1):1-51, July 2013.","title":"A survey of point-based POMDP solvers","authors":"Shani","year":"2013","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. Advances in neural information processing systems, 23, 2010.","title":"Monte-Carlo planning in large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Smith and Simmons, 2005] Trey Smith and Reid Simmons. Point-based POMDP algorithms: Improved analysis and implementation. In Proceedings of the Twenty-First Con- ference on Uncertainty in Artificial Intelligence, pages 542-549, 2005.","title":"Trey Smith and Reid Simmons. Point-based POMDP algorithms: Improved analysis and implementation","authors":"Simmons Smith","year":"2005","venue":"Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Smith, 2007] Trey Smith. Probabilistic Planning for Robotic Exploration. Carnegie Mellon University, 2007.","title":"Probabilistic Planning for Robotic Exploration","authors":"Trey Smith","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Tsang et al., 2022] Florence Tsang, Tristan Walker, Ryan A. MacDonald, Armin Sadeghi, and Stephen L. Smith. LAMP: Learning a Motion Policy to Repeat- edly Navigate in an Uncertain Environment. IEEE Transactions on Robotics, 38(3):1638-1652, June 2022.","title":"LAMP: Learning a Motion Policy to Repeatedly Navigate in an Uncertain Environment","authors":"Tsang","year":"2022","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Watkins, 1989] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, Cambridge University, 1989.","title":"Learning from delayed rewards","authors":"Christopher John, Cornish Hellaby Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wray and Zilberstein, 2019] Kyle Hollins Wray and Shlomo Zilberstein. Generalized Controllers in POMDP Decision-Making. In 2019 International Conference on Robotics and Automation (ICRA), pages 7166-7172, Montreal, QC, Canada, May 2019. IEEE.","title":"Generalized Controllers in POMDP Decision-Making","authors":"Zilberstein Wray, Kyle Hollins Wray, Shlomo Zilberstein","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}