{"type":"ScholarlyArticle","title":"Solving Long-run Average Reward Robust MDPs via Stochastic Games","authors":[{"name":"Krishnendu Chatterjee","slug":"krishnendu-chatterjee","given":"Krishnendu","family":"Chatterjee","affiliation":"Institute of Science and Technology Austria (ISTA) , Austria"},{"name":"Ehsan Kafshdar Goharshady","slug":"ehsan-kafshdar-goharshady","given":"Ehsan","family":"Kafshdar Goharshady","affiliation":"Institute of Science and Technology Austria (ISTA) , Austria"},{"name":"Mehrdad Karrabi","slug":"mehrdad-karrabi","given":"Mehrdad","family":"Karrabi","affiliation":"Institute of Science and Technology Austria (ISTA) , Austria"},{"name":"Petr Novotný","slug":"petr-novotny","given":"Petr","family":"Novotný","affiliation":"Masaryk University , Czech Republic; Institute of Science and Technology Austria (ISTA) ."},{"name":"Đorđe Žikelić","slug":"or-e-zikelic","given":"Đorđe","family":"Žikelić","affiliation":"Singapore Management University , Singapore"}],"abstract":"Markov decision processes (MDPs) provide a standard framework for sequential decision making under uncertainty. However, MDPs do not take uncertainty in transition probabilities into account. Robust Markov decision processes (RMDPs) address this shortcoming of MDPs by assigning to each transition an uncertainty set rather than a single probability value. In this work, we consider polytopic RMDPs in which all uncertainty sets are polytopes and study the problem of solving long-run average reward polytopic RMDPs. We present a novel perspective on this problem and show that it can be reduced to solving long-run average reward turn-based stochastic games with finite state and action spaces. This reduction allows us to derive several important consequences that were hitherto not known to hold for polytopic RMDPs. First, we derive new computational complexity bounds for solving long-run average reward polytopic RMDPs, showing for the first time that the threshold decision problem for them is in NP and coNP and that they admit a randomized algorithm with sub-exponential expected runtime. Second, we present Robust Polytopic Policy Iteration (RPPI), a novel policy iteration algorithm for solving long-run average reward polytopic RMDPs. Our experimental evaluation shows that RPPI is much more efficient in solving long-run average reward polytopic RMDPs compared to state-of-the-art methods based on value iteration.","doi":"10.24963/ijcai.2024/741","url":"https://ijcai.org/proceedings/2024/741","pdf":"https://ijcai.org/proceedings/2024/0741.pdf","markdown":"https://ijcai.org/proceedings/2024/741.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/741","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Planning and Scheduling","track":"Main Track","number":741,"pages":{"first":6707,"last":6715},"keywords":["Planning and Scheduling: PS: Markov decisions processes","Planning and Scheduling: PS: Theoretical foundations of planning","Agent-based and Multi-agent Systems: MAS: Formal verification, validation and synthesis","Planning and Scheduling: PS: Planning under uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Andersson and Miltersen, 2009] Daniel Andersson and Pe- ter Bro Miltersen. The complexity of solving stochastic games on graphs. In Yingfei Dong, Ding-Zhu Du, and Oscar Ibarra, editors, Algorithms and Computation, pages 112-121, Berlin, Heidelberg, 2009. Springer Berlin Hei- delberg.","title":"The complexity of solving stochastic games on graphs","authors":"Andersson, Miltersen, Daniel Andersson, Peter Bro Miltersen","year":"2009","venue":"Algorithms and Computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertrand et al., 2023] Nathalie Bertrand, Patricia Bouyer- Decitre, Nathanaël Fijalkow, and Mateusz Skomra. Stochastic games. In Nathanaël Fijalkow, editor, Games on Graphs. 2023.","title":"Stochastic games","authors":"Nathalie Bertrand, Patricia Bouyer-Decitre, Nathanaël Fijalkow, Mateusz Skomra","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chatterjee et al., 2023a] Krishnendu Chatterjee, Ehsan Kaf- shdar Goharshady, Mehrdad Karrabi, Petr Novotný, and Dorde Zikelic. Solving long-run average reward robust mdps via stochastic games. CoRR, abs/2312.13912, 2023.","title":"Solving long-run average reward robust mdps via stochastic games","authors":"Chatterjee","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chatterjee et al., 2023b] Krishnendu Chatterjee, Tobias Meggendorfer, Raimundo Saona, and Jakub Svoboda. Faster algorithm for turn-based stochastic games with bounded treewidth. In SODA, pages 4590-4605. SIAM, 2023.","title":"Faster algorithm for turn-based stochastic games with bounded treewidth","authors":"Chatterjee","year":"2023","venue":"SODA","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Condon, 1992] A. Condon. The Complexity of Stochastic Games. Information and Computation, 96(2):203-224, 1992.","title":"The Complexity of Stochastic Games","authors":"A Condon","year":"1992","venue":"Information and Computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Filar and Vrieze, 1996] J. Filar and K. Vrieze. Competitive Markov Decision Processes. 1996. [Ghavamzadeh et al., 2016]","title":"Competitive Markov Decision Processes","authors":"J Filar, K Vrieze","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow. Safe policy improve- ment by minimizing robust baseline regret. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, De- cember 5-10, 2016, Barcelona, Spain, pages 2298-2306, 2016.","title":"Safe policy improvement by minimizing robust baseline regret","authors":"Mohammad Ghavamzadeh, Marek Petrik, Yinlam Chow","year":"2016","venue":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gillette, 1957] G. Gillette. Stochastic Games with Zero Stop Probabilities. Contributions to the Theory of Games, vol III, pages 179-187, 1957.","title":"Stochastic Games with Zero Stop Probabilities","authors":"G Gillette","year":"1957","venue":"Contributions to the Theory of Games","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Givan et al., 2000] Robert Givan, Sonia M. Leach, and Thomas L. Dean. Bounded-parameter markov decision processes. Artif. Intell., 122(1-2):71-109, 2000.","title":"Bounded-parameter markov decision processes","authors":"Robert Givan, Sonia M Leach, Thomas L Dean","year":"2000","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goyal and Grand-Clément, 2023] Vineet Goyal and Julien Grand-Clément. Robust markov decision processes: Be- yond rectangularity. Math. Oper. Res., 48(1):203-226, 2023.","title":"Robust markov decision processes: Beyond rectangularity","authors":"Vineet Goyal, Julien Grand-Clément","year":"2023","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grand-Clément and Petrik, 2023] Julien Grand-Clément and Marek Petrik. Reducing blackwell and average optimality to discounted mdps via the blackwell discount factor. CoRR, abs/2302.00036, 2023.","title":"Reducing blackwell and average optimality to discounted mdps via the blackwell discount factor","authors":"Grand-Clément, Petrik ; Julien Grand-Clément, Marek Petrik","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Grand-Clement et al., 2023] Julien Grand-Clement, Marek Petrik, and Nicolas Vieille. Beyond discounted returns: Robust markov decision processes with average and black- well optimality. arXiv preprint arXiv:2312.03618, 2023.","title":"Beyond discounted returns: Robust markov decision processes with average and blackwell optimality","authors":"Julien Grand-Clement, Marek Petrik, Nicolas Vieille","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hansen et al., 2013] Thomas Dueholm Hansen, Peter Bro Miltersen, and Uri Zwick. Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor. J. ACM, 60(1):1:1-1:16, 2013.","title":"Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor","authors":"Thomas Dueholm Hansen, Peter Bro Miltersen, Uri Zwick","year":"2013","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hensel et al., 2022] Christian Hensel, Sebastian Junges, Joost-Pieter Katoen, Tim Quatmann, and Matthias Volk. The probabilistic model checker storm. Int. J. Softw. Tools Technol. Transf., 24(4):589-610, 2022.","title":"The probabilistic model checker storm","authors":"Hensel","year":"2022","venue":"Int. J. Softw. Tools Technol. Transf","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ho et al., 2018] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust mdps. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learn- ing, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1984-1993. PMLR, 2018.","title":"Fast bellman updates for robust mdps","authors":"Chin Pang Ho, Marek Petrik, Wolfram Wiesemann","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ho et al., 2021] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for l1-robust markov decision processes. J. Mach. Learn. Res., 22:275:1- 275:46, 2021.","title":"Partial policy iteration for l1-robust markov decision processes","authors":"Chin Pang Ho, Marek Petrik, Wolfram Wiesemann","year":"2021","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Iyengar, 2005] Garud N. Iyengar. Robust dynamic program- ming. Math. Oper. Res., 30(2):257-280, 2005.","title":"Robust dynamic programming","authors":"N Garud, Iyengar","year":"2005","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kaufman and Schaefer, 2013] David L. Kaufman and An- drew J. Schaefer. Robust modified policy iteration. IN- FORMS J. Comput., 25(3):396-410, 2013.","title":"Robust modified policy iteration","authors":"David L Kaufman, Andrew J Schaefer","year":"2013","venue":"IN-FORMS J. Comput","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kučera, 2011] A. Kučera. Turn-Based Stochastic Games. In K.R. Apt, E. Grädel (Eds.): Lectures in Game Theory for Computer Scientists, pages 146-184. Cambridge Univer- sity Press, 2011.","title":"Turn-Based Stochastic Games","authors":"A Kučera","year":"2011","venue":"Lectures in Game Theory for Computer Scientists","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liggett and Lippman, 1969] T.M. Liggett and S.A. Lipp- man. Stochastic Games with Perfect Information and Time Average Payoff. SIAM Review, 11(4):604-607, 1969.","title":"Stochastic Games with Perfect Information and Time Average Payoff","authors":"T M Liggett, S A Lippman","year":"1969","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ludwig, 1995] W. Ludwig. A Subexponential Random- ized Algorithm for the Simple Stochastic Game Problem. 117(1):151-155, 1995.","title":"A Subexponential Randomized Algorithm for the Simple Stochastic Game Problem","authors":"W Ludwig","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nilim and Ghaoui, 2003] Arnab Nilim and Laurent El Ghaoui. Robustness in markov decision problems with uncertain transition matrices. In Sebastian Thrun, Lawrence K. Saul, and Bernhard Schölkopf, editors, Ad- vances in Neural Information Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, December 8-13, 2003, Vancouver and Whistler, British Columbia, Canada], pages 839-846. MIT Press, 2003.","title":"Robustness in markov decision problems with uncertain transition matrices","authors":"Arnab Nilim, Laurent El Ghaoui","year":"2003","venue":"Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems, NIPS 2003","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":"Wiley Series in Probability and Statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Roy et al., 2017] Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Informa- tion Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3043-3052, 2017.","title":"Reinforcement learning under model mismatch","authors":"Roy","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shapley, 1953] L.S. Shapley. Stochastic Games. Proceed- ings of the National Academy of Sciences, 39:1095-1100, 1953.","title":"Stochastic Games","authors":"L S Shapley","year":"1953","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sinha and Ghate, 2016] Saumya Sinha and Archis Ghate. Policy iteration for robust nonstationary markov decision processes. Optim. Lett., 10(8):1613-1628, 2016.","title":"Policy iteration for robust nonstationary markov decision processes","authors":"Ghate Sinha, Saumya Sinha, Archis Ghate","year":"2016","venue":"Optim. Lett","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutton et al., 1999] Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller, editors, Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 -December 4, 1999], pages 1057-1063. The MIT Press, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David A Mcallester, Satinder Singh, Yishay Mansour","year":"1999","venue":"Advances in Neural Information Processing Systems 12","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tamar et al., 2014] Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approxima- tion. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 181-189. JMLR.org, 2014.","title":"Scaling up robust mdps using function approximation","authors":"Tamar","year":"2014","venue":"Proceedings of the 31th International Conference on Machine Learning, ICML 2014","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tessler et al., 2019] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In Kamalika Chaud- huri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Re- search, pages 6215-6224. PMLR, 2019.","title":"Action robust reinforcement learning and applications in continuous control","authors":"Chen Tessler, Yonathan Efroni, Shie Mannor","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning, ICML 2019","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tewari and Bartlett, 2007] Ambuj Tewari and Peter L. Bartlett. Bounded parameter markov decision processes with average reward criterion. In Nader H. Bshouty and Claudio Gentile, editors, Learning Theory, 20th Annual Conference on Learning Theory, COLT 2007, San Diego, CA, USA, June 13-15, 2007, Proceedings, volume 4539 of Lecture Notes in Computer Science, pages 263-277. Springer, 2007.","title":"Bounded parameter markov decision processes with average reward criterion","authors":"Ambuj Tewari, Peter L Bartlett","year":"2007","venue":"Learning Theory, 20th Annual Conference on Learning Theory, COLT 2007","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Towers et al., 2023] Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023.","title":null,"authors":"Mark Towers, Jordan K Terry, Ariel Kwiatkowski, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, K G Arjun, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan, Jin Shen, Omar G Younis, Gymnasium","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang and Zou, 2021] Yue Wang and Shaofeng Zou. On- line robust reinforcement learning with model uncertainty. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems 34: Annual Conference on Neural Information Pro- cessing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 7193-7206, 2021.","title":"Online robust reinforcement learning with model uncertainty","authors":"Yue Wang, Shaofeng Zou","year":"2021","venue":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang and Zou, 2022] Yue Wang and Shaofeng Zou. Pol- icy gradient method for robust reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, Inter- national Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 23484-23526. PMLR, 2022.","title":"Policy gradient method for robust reinforcement learning","authors":"Yue Wang, Shaofeng Zou","year":"2022","venue":"International Conference on Machine Learning, ICML 2022","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2023] Yue Wang, Alvaro Velasquez, George K. Atia, Ashley Prater-Bennette, and Shaofeng Zou. Ro- bust average-reward markov decision processes. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Appli- cations of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelli- gence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 15215-15223. AAAI Press, 2023.","title":"Robust average-reward markov decision processes","authors":"Yue Wang, Alvaro Velasquez, George K Atia, Ashley Prater-Bennette, Shaofeng Zou","year":"2023","venue":"Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wiesemann et al., 2013] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Math. Oper. Res., 38(1):153-183, 2013.","title":"Robust markov decision processes","authors":"Wiesemann","year":"2013","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang et al., 2022] Wenhao Yang, Liangyu Zhang, and Zhi- hua Zhang. Toward theoretical understandings of ro- bust markov decision processes: Sample complexity and asymptotics. The Annals of Statistics, 50(6):3223-3248, 2022.","title":"Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics","authors":"Wenhao Yang, Liangyu Zhang, Zhihua Zhang","year":"2022","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}