{"type":"ScholarlyArticle","title":"Robust Policy Synthesis for Uncertain POMDPs via Convex Optimization","authors":[{"name":"Marnix Suilen","slug":"marnix-suilen","given":"Marnix","family":"Suilen","affiliation":"Department of Software Science , Radboud University , The Netherlands"},{"name":"Nils Jansen","slug":"nils-jansen","given":"Nils","family":"Jansen","affiliation":"Department of Software Science , Radboud University , The Netherlands"},{"name":"Murat Cubuktepe","slug":"murat-cubuktepe","given":"Murat","family":"Cubuktepe","affiliation":"Department of Aerospace Engineering and Engineering Mechanics , University of Texas at Austin , USA"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"Department of Aerospace Engineering and Engineering Mechanics , University of Texas at Austin , USA"}],"abstract":"We study the problem of policy synthesis for uncertain partially observable Markov decision processes (uPOMDPs).\r\nThe transition probability function of uPOMDPs is only known to belong to a so-called uncertainty set, for instance in the form of probability intervals.\r\nSuch a model arises when, for example, an agent operates under information limitation due to imperfect knowledge about the accuracy of its sensors.\r\nThe goal is to compute a policy for the agent that is robust against all possible probability distributions within the uncertainty set.\r\nIn particular, we are interested in a policy that robustly ensures the satisfaction of temporal logic and expected reward specifications.\r\nWe state the underlying optimization problem as a semi-infinite quadratically-constrained quadratic program (QCQP), which has finitely many variables and infinitely many constraints.\r\nSince QCQPs are non-convex in general and practically infeasible to solve, we resort to the so-called convex-concave procedure to convexify the QCQP.\r\nEven though convex, the resulting optimization problem still has infinitely many constraints and is NP-hard.\r\nFor uncertainty sets that form convex polytopes, we provide a transformation of the problem to a convex QCQP with finitely many constraints. \r\nWe demonstrate the feasibility of our approach by means of several case studies that highlight typical bottlenecks for our problem. \r\nIn particular, we show that we are able to solve benchmarks with hundreds of thousands of states, hundreds of different observations, and we investigate the effect of different levels of uncertainty in the models.","doi":"10.24963/ijcai.2020/569","url":"https://ijcai.org/proceedings/2020/569","pdf":"https://ijcai.org/proceedings/2020/0569.pdf","markdown":"https://ijcai.org/proceedings/2020/569.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/569","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Planning and Scheduling","track":"Main track","number":569,"pages":{"first":4113,"last":4120},"keywords":["Planning and Scheduling: POMDPs","Uncertainty in AI: Markov Decision Processes","Planning and Scheduling: Planning under Uncertainty"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25369/file_25369.m3u8","length_sec":190,"provider":"hls"}],"references":[{"position":1,"raw":"[Ahmadi et al., 2018] Mohamadreza Ahmadi, Murat Cubuk- tepe, Nils Jansen, and Ufuk Topcu. Verification of Uncer- tain POMDPs Using Barrier Certificates. In Allerton, pages 115-122. IEEE, 2018.","title":"Verification of Uncertain POMDPs Using Barrier Certificates","authors":"Ahmadi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amato et al., 2006] Christopher Amato, Daniel S Bernstein, and Shlomo Zilberstein. Solving POMDPs Using Quadrat- ically Constrained Linear Programs. In AAMAS, pages 341-343. ACM, 2006.","title":"Solving POMDPs Using Quadratically Constrained Linear Programs","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2006","venue":"AAMAS","doi":null,"url":null,"internal_url":"/proceedings/2007/389"},{"position":3,"raw":"[Aras et al., 2007] Raghav Aras, Alain Dutech, and Franc ¸ois Charpillet. Mixed Integer Linear Programming for Exact Finite-Horizon Planning in Decentralized POMDPs. In ICAPS, pages 18-25. AAAI, 2007.","title":"Mixed Integer Linear Programming for Exact Finite-Horizon Planning in Decentralized POMDPs","authors":"Raghav Aras, Alain Dutech, Franc ¸ois, Charpillet","year":"2007","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bagnell et al., 2001] J Andrew Bagnell, Andrew Y Ng, and Jeff G Schneider. Solving Uncertain Markov Decision Processes. 2001.","title":"Solving Uncertain Markov Decision Processes","authors":"Andrew Bagnell, Andrew Y Ng, Jeff G Schneider","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Ka- toen. Principles of Model Checking. MIT Press, 2008.","title":"Principles of Model Checking","authors":"Christel Baier, Joost-Pieter Katoen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Benedikt et al., 2013] Michael Benedikt, Rastislav Lenhardt, and James Worrell. LTL Model Checking of Interval Markov Chains. In TACAS, pages 32-46. Springer, 2013.","title":"LTL Model Checking of Interval Markov Chains","authors":"Michael Benedikt, Rastislav Lenhardt, James Worrell","year":"2013","venue":"TACAS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bertsimas et al., 2011] Dimitris Bertsimas, David B Brown, and Constantine Caramanis. Theory and Applications of Robust Optimization. SIAM review, 53(3):464-501, 2011.","title":"Theory and Applications of Robust Optimization","authors":"Dimitris Bertsimas, David B Brown, Constantine Caramanis","year":"2011","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bouton et al., 2020] Maxime Bouton, Jana Tumova, and Mykel J. Kochenderfer. Point-Based Methods for Model Checking in Partially Observable Markov Decision Pro- cesses. CoRR, abs/2001.03809, 2020.","title":"Point-Based Methods for Model Checking in Partially Observable Markov Decision Processes","authors":"Maxime Bouton, Jana Tumova, Mykel J Kochenderfer","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Burns and Brock, 2007] Brendan Burns and Oliver Brock. Sampling-Based Motion Planning with Sensing Uncer- tainty. In ICRA, pages 3313-3318. IEEE, 2007.","title":"Sampling-Based Motion Planning with Sensing Uncertainty","authors":"Brock ; Burns, Brendan Burns, Oliver Brock","year":"2007","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Carr et al., 2019] Steven Carr, Nils Jansen, Ralf Wimmer, Alexandru Constantin Serban, Bernd Becker, and Ufuk Topcu. Counterexample-guided strategy improvement for pomdps using recurrent neural networks. In IJCAI, pages 5532-5539. ijcai.org, 2019.","title":"Counterexample-guided strategy improvement for pomdps using recurrent neural networks","authors":"Steven Carr, Nils Jansen, Ralf Wimmer, Alexandru Constantin Serban, Bernd Becker, Ufuk Topcu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/768"},{"position":11,"raw":"[Chamie and Mostafa, 2018] Mahmoud El Chamie and Hala Mostafa. Robust Action Selection in Partially Observable Markov Decision Processes with Model Uncertainty. In CDC, pages 5586-5591. IEEE, 2018.","title":"Robust Action Selection in Partially Observable Markov Decision Processes with Model Uncertainty","authors":"Mostafa Chamie, Mahmoud El Chamie, Hala Mostafa","year":"2018","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chatterjee et al., 2016] Krishnendu Chatterjee, Martin Chmelík, Raghav Gupta, and Ayush Kanodia. Optimal Cost Almost-sure Reachability in POMDPs. Artif. Intell., 234:26-48, 2016.","title":"Optimal Cost Almost-sure Reachability in POMDPs","authors":"Krishnendu Chatterjee, Martin Chmelík, Raghav Gupta, Ayush Kanodia","year":"2016","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2017] Robert S. Chen, Brendan Lucier, Yaron Singer, and Vasilis Syrgkanis. Robust Optimization for Non-Convex Objectives. In NIPS, pages 4705-4714, 2017.","title":"Robust Optimization for Non-Convex Objectives","authors":"Robert S Chen, Brendan Lucier, Yaron Singer, Vasilis Syrgkanis","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Cubuktepe et al., 2018] Murat Cubuktepe, Nils Jansen, Se- bastian Junges, Joost-Pieter Katoen, and Ufuk Topcu. Syn- thesis in pmdps: A tale of 1001 parameters. In ATVA, volume 11138 of LNCS, pages 160-176. Springer, 2018.","title":"Synthesis in pmdps: A tale of 1001 parameters","authors":"Nils Murat Cubuktepe, Sebastian Jansen, Joost-Pieter Junges, Ufuk Katoen, Topcu","year":"2018","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Cubuktepe et al., 2020] Murat Cubuktepe, Nils Jansen, Se- bastian Junges, Joost-Pieter Katoen, and Ufuk Topcu. Scenario-based verification of uncertain mdps. In TACAS (1), volume 12078 of LNCS, pages 287-305. Springer, 2020.","title":"Scenario-based verification of uncertain mdps","authors":"Nils Murat Cubuktepe, Sebastian Jansen, Joost-Pieter Junges, Ufuk Katoen, Topcu","year":"2020","venue":"TACAS (1)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dehnert et al., 2015] Christian Dehnert, Sebastian Junges, Nils Jansen, Florian Corzilius, Matthias Volk, Harold Bruin- tjes, Joost-Pieter Katoen, and Erika Ábrahám. PROPhESY: A probabilistic parameter synthesis tool. In CAV, volume 9206 of LNCS, pages 214-231. Springer, 2015.","title":"PROPhESY: A probabilistic parameter synthesis tool","authors":"Dehnert","year":"2015","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Dehnert et al., 2017] Christian Dehnert, Sebastian Junges, Joost-Pieter Katoen, and Matthias Volk. A Storm is Com- ing: A Modern Probabilistic Model Checker. In CAV (2), volume 10427 of LNCS, pages 592-600. Springer, 2017.","title":"A Storm is Coming: A Modern Probabilistic Model Checker","authors":"Dehnert","year":"2017","venue":"CAV (2)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Fukuda and Prodon, 1996] Komei Fukuda and Alain Prodon. Double Description Method Revisited. In Combinatorics and Computer Science, pages 91-111, Berlin, Heidelberg, 1996. Springer.","title":"Double Description Method Revisited","authors":"Prodon Fukuda, Komei Fukuda, Alain Prodon","year":"1996","venue":"Combinatorics and Computer Science","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Givan et al., 2000] Robert Givan, Sonia Leach, and Thomas Dean. Bounded-Parameter Markov Decision Processes. Artif. Intell., 122(1-2):71-109, 2000.","title":"Bounded-Parameter Markov Decision Processes","authors":"Robert Givan, Sonia Leach, Thomas Dean","year":"2000","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hahn et al., 2017] Ernst Moritz Hahn, Vahid Hashemi, Hol- ger Hermanns, Morteza Lahijanian, and Andrea Turrini. Multi-Objective Robust Strategy Synthesis for Interval Markov Decision Processes. In QEST, volume 10503 of LNCS, pages 207-223. Springer, 2017.","title":"Multi-Objective Robust Strategy Synthesis for Interval Markov Decision Processes","authors":"Hahn","year":"2017","venue":"QEST","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Itoh and Nakamura, 2007] Hideaki Itoh and Kiyohiko Naka- mura. Partially Observable Markov Decision Processes with Imprecise Parameters. Artif. Intell., 171(8):453 -490, 2007.","title":"Partially Observable Markov Decision Processes with Imprecise Parameters","authors":"Nakamura Itoh, Hideaki Itoh, Kiyohiko Nakamura","year":"2007","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Junges et al., 2018] Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, and Bernd Becker. Finite-State Controllers of POMDPs using Parameter Synthesis. In UAI, pages 519- 529, 2018.","title":"Finite-State Controllers of POMDPs using Parameter Synthesis","authors":"Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, Bernd Becker","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Junges et al., 2019] Sebastian Junges, Erika Ábrahám, Chris- tian Hensel, Nils Jansen, Joost-Pieter Katoen, Tim Quat- mann, and Matthias Volk. Parameter synthesis for markov models. CoRR, abs/1903.07993, 2019.","title":"Parameter synthesis for markov models","authors":"Sebastian Junges, Erika Ábrahám, Christian Hensel, Nils Jansen, Joost-Pieter Katoen, Tim Quatmann, Matthias Volk","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artif. Intell., 1998.","title":"Planning and Acting in Partially Observable Stochastic Domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kochenderfer, 2015] Mykel J Kochenderfer. Decision Mak- ing Under Uncertainty: Theory and Application. 2015.","title":"Decision Making Under Uncertainty: Theory and Application","authors":"J Mykel, Kochenderfer","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kumar et al., 2016] Akshat Kumar, Hala Mostafa, and Shlomo Zilberstein. Dual Formulations for Optimizing Dec-POMDP Controllers. In ICAPS, pages 202-210. AAAI Press, 2016.","title":"Dual Formulations for Optimizing Dec-POMDP Controllers","authors":"Akshat Kumar, Hala Mostafa, Shlomo Zilberstein","year":"2016","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lipp and Boyd, 2016] Thomas Lipp and Stephen Boyd. Vari- ations and Extension of the Convex-Concave Procedure. Optimization and Engineering, 17(2):263-287, 2016.","title":"Thomas Lipp and Stephen Boyd. Variations and Extension of the Convex-Concave Procedure","authors":"Boyd Lipp","year":"2016","venue":"Optimization and Engineering","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Littman et al., 1995] Michael L Littman, Anthony R Cas- sandra, and Leslie Pack Kaelbling. Learning Policies for Partially Observable Environments: Scaling up. In ICML, pages 362-370. Elsevier, 1995.","title":"Learning Policies for Partially Observable Environments: Scaling up","authors":"Anthony R Michael L Littman, Leslie Cassandra, Kaelbling Pack","year":"1995","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Löfberg, 2012] Johan Löfberg. Automatic Robust Con- vex Programming. Optimization Methods and Software, 27(1):115-129, 2012.","title":"Automatic Robust Convex Programming","authors":"Johan Löfberg","year":"2012","venue":"Optimization Methods and Software","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Madani et al., 1999] Omid Madani, Steve Hanks, and Anne Condon. On the Undecidability of Probabilistic Planning and Infinite-Horizon Partially Observable Markov Decision Problems. In AAAI, pages 541-548. AAAI Press, 1999.","title":"On the Undecidability of Probabilistic Planning and Infinite-Horizon Partially Observable Markov Decision Problems","authors":"Omid Madani, Steve Hanks, Anne Condon","year":"1999","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[McCallum, 1993] R Andrew McCallum. Overcoming In- complete Perception With Utile Distinction Memory. 1993.","title":"Overcoming Incomplete Perception With Utile Distinction Memory","authors":"Andrew Mccallum","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Meuleu et al., 2010] N. Meuleu, C. Plaunt, D. E. Smith, and T. Smith. A POMDP for Optimal Motion Planning with Uncertain Dynamics. In ICAPS: POMDP Practitioners Workshop, 2010.","title":"A POMDP for Optimal Motion Planning with Uncertain Dynamics","authors":"N Meuleu, C Plaunt, D E Smith, T Smith","year":"2010","venue":"ICAPS: POMDP Practitioners Workshop","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Nilim and Ghaoui, 2005] Arnab Nilim and Laurent El Ghaoui. Robust Control of Markov Decision Processes with Uncertain Transition Matrices. Operations Research, 53(5):780-798, 2005.","title":"Robust Control of Markov Decision Processes with Uncertain Transition Matrices","authors":"Arnab Nilim, Laurent El Ghaoui","year":"2005","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Norman et al., 2017] Gethin Norman, David Parker, and Xueyi Zou. Verification and Control of Partially Observable Probabilistic Systems. Real-Time Systems, 53(3):354-402, 2017.","title":"Verification and Control of Partially Observable Probabilistic Systems","authors":"Gethin Norman, David Parker, Xueyi Zou","year":"2017","venue":"Real-Time Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Osogami, 2015] Takayuki Osogami. Robust partially observ- able markov decision process. In ICML, volume 37, pages 106-115, 2015.","title":"Robust partially observable markov decision process","authors":"Takayuki Osogami","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Park and Boyd, 2017] Jaehyun Park and Stephen Boyd. Gen- eral Heuristics for Nonconvex Quadratically Constrained Quadratic Programming. arXiv preprint arXiv:1703.07870, 2017.","title":"Jaehyun Park and Stephen Boyd","authors":"Boyd Park","year":"2017","venue":"General Heuristics for Nonconvex Quadratically Constrained Quadratic Programming","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Pnueli, 1977] Amir Pnueli. The Temporal Logic of Pro- grams. In FOCS, pages 46-57. IEEE Computer Society, 1977.","title":"The Temporal Logic of Programs","authors":"Amir Pnueli, Pnueli","year":"1977","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Puggelli et al., 2013] Alberto Puggelli, Wenchao Li, Al- berto L. Sangiovanni-Vincentelli, and Sanjit A. Seshia. Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties. In CAV, volume 8044 of LNCS, pages 527-542. Springer, 2013.","title":"Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties","authors":"Alberto Puggelli, Wenchao Li, Alberto L Sangiovanni-Vincentelli, Sanjit A Seshia","year":"2013","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Vlassis et al., 2012] Nikos Vlassis, Michael L. Littman, and David Barber. On the Computational Complexity of Stochastic Controller Optimization in POMDPs. ACM Trans. on Computation Theory, 4(4):12:1-12:8, 2012.","title":"On the Computational Complexity of Stochastic Controller Optimization in POMDPs","authors":"Nikos Vlassis, Michael L Littman, David Barber","year":"2012","venue":"ACM Trans. on Computation Theory","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wiesemann et al., 2013] Wolfram Wiesemann, Daniel Kuhn, and Berc ¸Rustem. Robust Markov Decision Processes. Mathematics of Operations Research, 38(1):153-183, 2013.","title":"Robust Markov Decision Processes","authors":"Wiesemann","year":"2013","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Winkler et al., 2019] Tobias Winkler, Sebastian Junges, Guillermo A. Pérez, and Joost-Pieter Katoen. On the com- plexity of reachability in parametric markov decision pro- cesses. In CONCUR, volume 140 of LIPIcs, pages 14:1- 14:17. Schloss Dagstuhl -Leibniz-Zentrum für Informatik, 2019.","title":"On the complexity of reachability in parametric markov decision processes","authors":"Winkler","year":"2019","venue":"CONCUR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wolff et al., 2012] Eric M. Wolff, Ufuk Topcu, and Richard M. Murray. Robust Control of Uncertain Markov Decision Processes with Temporal Logic Specifications. In CDC, pages 3372-3379. IEEE, 2012.","title":"Robust Control of Uncertain Markov Decision Processes with Temporal Logic Specifications","authors":"Eric M Wolff, Ufuk Topcu, Richard M Murray","year":"2012","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yang et al., 2011] Lei Yang, Sugumar Murugesan, and Jun- shan Zhang. Real-time Scheduling over Markovian Chan- nels: When Partial Observability Meets Hard Deadlines. In GLOBECOM, pages 1-5. IEEE, 2011.","title":"Real-time Scheduling over Markovian Channels: When Partial Observability Meets Hard Deadlines","authors":"Yang","year":"2011","venue":"GLOBECOM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}