{"type":"ScholarlyArticle","title":"Imprecise Probabilities Meet Partial Observability: Game Semantics for Robust POMDPs","authors":[{"name":"Eline M. Bovy","slug":"eline-m-bovy","given":"Eline M.","family":"Bovy","affiliation":"Radboud University , The Netherlands"},{"name":"Marnix Suilen","slug":"marnix-suilen","given":"Marnix","family":"Suilen","affiliation":"Radboud University , The Netherlands"},{"name":"Sebastian Junges","slug":"sebastian-junges","given":"Sebastian","family":"Junges","affiliation":"Radboud University , The Netherlands"},{"name":"Nils Jansen","slug":"nils-jansen","given":"Nils","family":"Jansen","affiliation":"Radboud University , The Netherlands; Ruhr-University Bochum , Germany"}],"abstract":"Partially observable Markov decision processes (POMDPs) rely on the key assumption that probability distributions are precisely known. Robust POMDPs (RPOMDPs) alleviate this concern by defining imprecise probabilities, referred to as uncertainty sets. While robust MDPs have been studied extensively, work on RPOMDPs is limited and primarily focuses on algorithmic solution methods. We expand the theoretical understanding of RPOMDPs by showing that 1) different assumptions on the uncertainty sets affect optimal policies and values; 2) RPOMDPs have a  partially observable stochastic game (POSG) semantic; and 3) the same RPOMDP with different assumptions leads to semantically different POSGs and, thus, different policies and values. These novel semantics for RPOMDPs give access to results for POSGs, studied in game theory; concretely, we show the existence of a Nash equilibrium.  Finally, we classify the existing RPOMDP literature using our semantics, clarifying under which uncertainty assumptions these existing works operate.","doi":"10.24963/ijcai.2024/740","url":"https://ijcai.org/proceedings/2024/740","pdf":"https://ijcai.org/proceedings/2024/0740.pdf","markdown":"https://ijcai.org/proceedings/2024/740.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/740","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Planning and Scheduling","track":"Main Track","number":740,"pages":{"first":6697,"last":6706},"keywords":["Agent-based and Multi-agent Systems: MAS: Formal verification, validation and synthesis","Uncertainty in AI: UAI: Sequential decision making","Planning and Scheduling: PS: POMDPs"],"videos":[],"references":[{"position":1,"raw":"[Behzadian et al., 2021] Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for L ∞ -constrained s-rectangular robust MDPs. In NeurIPS, pages 25982- 25992, 2021.","title":"Fast algorithms for L ∞ -constrained s-rectangular robust MDPs","authors":"Bahram Behzadian, Marek Petrik, Chin Pang Ho","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bovy et al., 2024] Eline M. Bovy, Marnix Suilen, Sebastian Junges, and Nils Jansen. Imprecise probabilities meet par- tial observability: Game semantics for robust POMDPs. CoRR, abs/2405.04941, 2024.","title":"Imprecise probabilities meet partial observability: Game semantics for robust POMDPs","authors":"M Eline, Marnix Bovy, Sebastian Suilen, Nils Junges, Jansen","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bovy, 2023] Eline M. Bovy. The Underlying Belief Model of Uncertain Partially Observable Markov Decision Pro- cesses. Master thesis, Radboud University, 2023.","title":"The Underlying Belief Model of Uncertain Partially Observable Markov Decision Processes","authors":"M Eline, Bovy","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chamie and Mostafa, 2018] Mahmoud El Chamie and Hala Mostafa. Robust action selection in partially observable Markov decision processes with model uncertainty. In CDC, pages 5586-5591. IEEE, 2018.","title":"Robust action selection in partially observable Markov decision processes with model uncertainty","authors":"Mostafa Chamie, Mahmoud El Chamie, Hala Mostafa","year":"2018","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chatterjee et al., 2016] Krishnendu Chatterjee, Martin Chmelik, Raghav Gupta, and Ayush Kanodia. Optimal cost almost-sure reachability in POMDPs. Artif. Intell., 234:26-48, 2016.","title":"Optimal cost almost-sure reachability in POMDPs","authors":"Krishnendu Chatterjee, Martin Chmelik, Raghav Gupta, Ayush Kanodia","year":"2016","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chatterjee et al., 2023] Krishnendu Chatterjee, Ehsan Kaf- shdar Goharshady, Mehrdad Karrabi, Petr Novotnỳ, and Ðor de Žikelić. Solving long-run average reward robust MDPs via stochastic games. CoRR, abs/2312.13912, 2023.","title":"Solving long-run average reward robust MDPs via stochastic games","authors":"Krishnendu Chatterjee, Ehsan Kafshdar Goharshady, Mehrdad Karrabi, Petr Novotnỳ, Ðor De, Žikelić","year":"2023","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2024/741"},{"position":7,"raw":"[Cubuktepe et al., 2021] Murat Cubuktepe, Nils Jansen, Se- bastian Junges, Ahmadreza Marandi, Marnix Suilen, and Ufuk Topcu. Robust finite-state controllers for uncertain POMDPs. In AAAI, pages 11792-11800. AAAI Press, 2021.","title":"Robust finite-state controllers for uncertain POMDPs","authors":"Nils Murat Cubuktepe, Sebastian Jansen, Ahmadreza Junges, Marnix Marandi, Ufuk Suilen, Topcu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Delage et al., 2023] Aurélien Delage, Olivier Buffet, Jilles S. Dibangoye, and Abdallah Saffidine. HSVI can solve zero-sum partially observable stochastic games. Dynamic Games and Applications, 2023.","title":"HSVI can solve zero-sum partially observable stochastic games","authors":"Aurélien Delage, Olivier Buffet, Jilles S Dibangoye, Abdallah Saffidine","year":"2023","venue":"Dynamic Games and Applications","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fijalkow et al., 2023] Nathanaël Fijalkow, Nathalie Bertrand, Patricia Bouyer-Decitre, Romain Brenguier, Arnaud Carayol, John Fearnley, Hugo Gimbert, Florian Horn, Rasmus Ibsen-Jensen, Nicolas Markey, Benjamin Monmege, Petr Novotný, Mickael Randour, Ocan Sankur, Sylvain Schmitz, Olivier Serre, and Mateusz Skomra. Games on graphs. CoRR, abs/2305.10546, 2023.","title":"Games on graphs","authors":"Nathanaël Fijalkow, Nathalie Bertrand, Patricia Bouyer-Decitre, Romain Brenguier, Arnaud Carayol, John Fearnley, Hugo Gimbert, Florian Horn, Rasmus Ibsen-Jensen, Nicolas Markey, Benjamin Monmege, Petr Novotný, Mickael Randour, Ocan Sankur, Sylvain Schmitz, Olivier Serre, Mateusz Skomra","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gillette, 1957] Dean Gillette. Stochastic games with zero stop probabilities. Contributions to the Theory of Games, 3:179-187, 1957.","title":"Stochastic games with zero stop probabilities","authors":"Dean Gillette","year":"1957","venue":"Contributions to the Theory of Games","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grand-Clément et al., 2023] Julien Grand-Clément, Marek Petrik, and Nicolas Vieille. Beyond discounted re- turns: Robust Markov decision processes with average and blackwell optimality. CoRR, abs/2312.03618, 2023.","title":"Beyond discounted returns: Robust Markov decision processes with average and blackwell optimality","authors":"Julien Grand-Clément, Marek Petrik, Nicolas Vieille","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ho et al., 2018] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust MDPs. In ICML, volume 80 of Proceedings of Machine Learning Research, pages 1984-1993. PMLR, 2018.","title":"Fast bellman updates for robust MDPs","authors":"Ho","year":"2018","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ho et al., 2021] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for L 1 -robust Markov decision processes. J. Mach. Learn. Res., 22:275:1- 275:46, 2021.","title":"Partial policy iteration for L 1 -robust Markov decision processes","authors":"Chin Pang Ho, Marek Petrik, Wolfram Wiesemann","year":"2021","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Iyengar, 2005] Garud N. Iyengar. Robust dynamic program- ming. Math. Oper. Res., 30(2):257-280, 2005.","title":"Robust dynamic programming","authors":"N Garud, Iyengar","year":"2005","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jaksch et al., 2010] Thomas Jaksch, Ronald Ortner, and Pe- ter Auer. Near-optimal regret bounds for reinforcement learning. J. Mach. Learn. Res., 11:1563-1600, 2010.","title":"Near-optimal regret bounds for reinforcement learning","authors":"Thomas Jaksch, Ronald Ortner, Peter Auer","year":"2010","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jansen et al., 2022] Nils Jansen, Sebastian Junges, and Joost-Pieter Katoen. Parameter synthesis in Markov mod- els: A gentle survey. In Principles of Systems Design, vol- ume 13660 of LNCS, pages 407-437. Springer, 2022.","title":"Parameter synthesis in Markov models: A gentle survey","authors":"Jansen","year":"2022","venue":"Principles of Systems Design","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., 101(1-2):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kwiatkowska et al., 2022] Marta Kwiatkowska, Gethin Norman, David Parker, Gabriel Santos, and Rui Yan. Probabilistic model checking for strategic equilibria- based decision making: Advances and challenges (invited talk). In MFCS, volume 241 of LIPIcs, pages 4:1-4:22. Schloss Dagstuhl -Leibniz-Zentrum für Informatik, 2022.","title":"Probabilistic model checking for strategic equilibriabased decision making: Advances and challenges (invited talk)","authors":"Marta Kwiatkowska, Gethin Norman, David Parker, Gabriel Santos, Rui Yan","year":"2022","venue":"MFCS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lopez and Vercher, 1986] M.A. Lopez and D.E. Vercher. Convex semi-infinite games. Journal of optimization the- ory and applications, 50(2):289-312, 1986.","title":"Convex semi-infinite games","authors":"M A Lopez, D E Vercher","year":"1986","venue":"Journal of optimization theory and applications","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Moos et al., 2022] Janosch Moos, Kay Hansel, Hany Ab- dulsamad, Svenja Stark, Debora Clever, and Jan Pe- ters. Robust reinforcement learning: A review of foun- dations and recent advances. Mach. Learn. Knowl. Extr., 4(1):276-315, 2022.","title":"Robust reinforcement learning: A review of foundations and recent advances","authors":"Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, Jan Peters","year":"2022","venue":"Mach. Learn. Knowl. Extr","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nakao et al., 2021] Hideaki Nakao, Ruiwei Jiang, and Siqian Shen. Distributionally robust partially observable Markov decision process with moment-based ambiguity. SIAM J. Optim., 31(1):461-488, 2021.","title":"Distributionally robust partially observable Markov decision process with moment-based ambiguity","authors":"Hideaki Nakao, Ruiwei Jiang, Siqian Shen","year":"2021","venue":"SIAM J. Optim","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nilim and Ghaoui, 2005] Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Oper. Res., 53(5):780-798, 2005.","title":"Robust control of Markov decision processes with uncertain transition matrices","authors":"Arnab Nilim, Laurent El Ghaoui","year":"2005","venue":"Oper. Res","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Osogami, 2015] Takayuki Osogami. Robust partially ob- servable Markov decision process. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 106-115. JMLR.org, 2015.","title":"Robust partially observable Markov decision process","authors":"Takayuki Osogami","year":"2015","venue":"JMLR Workshop and Conference Proceedings","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peters, 2015] Hans Peters. Game Theory: A Multi-Leveled Approach. Springer Texts in Business and Economics. Springer, second edition, 2015.","title":"Game Theory: A Multi-Leveled Approach","authors":"Hans Peters, Peters","year":"2015","venue":"Springer Texts in Business and Economics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Petrik and Subramanian, 2014] Marek Petrik and Dhar- mashankar Subramanian. RAAM: the benefits of robust- ness in approximating aggregated MDPs in reinforcement learning. In NIPS, pages 1979-1987, 2014.","title":"RAAM: the benefits of robustness in approximating aggregated MDPs in reinforcement learning","authors":"Subramanian Petrik, Marek Petrik, Dharmashankar Subramanian","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":"Wiley Series in Probability and Statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rasouli and Saghafian, 2018] Mohammad Rasouli and Soroush Saghafian. Robust partially observable Markov decision processes. HKS Working Paper, RWP18-027, 2018.","title":"Robust partially observable Markov decision processes","authors":"Mohammad Rasouli, Soroush Saghafian","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Saghafian, 2018] Soroush Saghafian. Ambiguous partially observable Markov decision processes: Structural results and applications. J. Econ. Theory, 178:1-35, 2018.","title":"Ambiguous partially observable Markov decision processes: Structural results and applications","authors":"Soroush Saghafian","year":"2018","venue":"J. Econ. Theory","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Spaan, 2012] Matthijs T. J. Spaan. Partially observable Markov decision processes. In Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization, pages 387-414. Springer, 2012.","title":"Partially observable Markov decision processes","authors":"T J Matthijs, Spaan","year":"2012","venue":"Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Suilen et al., 2020] Marnix Suilen, Nils Jansen, Murat Cubuktepe, and Ufuk Topcu. Robust policy synthesis for uncertain POMDPs via convex optimization. In IJCAI, pages 4113-4120. ijcai.org, 2020.","title":"Robust policy synthesis for uncertain POMDPs via convex optimization","authors":"Marnix Suilen, Nils Jansen, Murat Cubuktepe, Ufuk Topcu","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/569"},{"position":32,"raw":"[Suilen et al., 2022] Marnix Suilen, Thiago D. Simão, David Parker, and Nils Jansen. Robust anytime learning of Markov decision processes. In NeurIPS, 2022.","title":"Robust anytime learning of Markov decision processes","authors":"Marnix Suilen, D Thiago, David Simão, Nils Parker, Jansen","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2023] Qiuhao Wang, Chin Pang Ho, and Marek Petrik. Policy gradient in robust MDPs with global conver- gence guarantee. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 35763-35797. PMLR, 2023.","title":"Policy gradient in robust MDPs with global convergence guarantee","authors":"Wang","year":"2023","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wiesemann et al., 2013] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Math. Oper. Res., 38(1):153-183, 2013.","title":"Robust Markov decision processes","authors":"Wiesemann","year":"2013","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu and Mannor, 2010] Huan Xu and Shie Mannor. Distri- butionally robust Markov decision processes. In NIPS, pages 2505-2513.","title":"Distributionally robust Markov decision processes","authors":"Huan Xu, Shie Mannor","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"Curran Associates, Inc., 2010.","title":null,"authors":"Curran Associates, Inc","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}