{"type":"ScholarlyArticle","title":"Certified Policy Verification and Synthesis for MDPs under Distributional Reach-Avoidance Properties","authors":[{"name":"S. Akshay","slug":"s-akshay","given":"S.","family":"Akshay","affiliation":"Indian Institute of Technology Bombay , India"},{"name":"Krishnendu Chatterjee","slug":"krishnendu-chatterjee","given":"Krishnendu","family":"Chatterjee","affiliation":"Institute of Science and Technology Austria (ISTA) , Austria"},{"name":"Tobias Meggendorfer","slug":"tobias-meggendorfer","given":"Tobias","family":"Meggendorfer","affiliation":"Lancaster University Leipzig , Germany"},{"name":"Đorđe Žikelić","slug":"or-e-zikelic","given":"Đorđe","family":"Žikelić","affiliation":"Singapore Management University , Singapore"}],"abstract":"Markov Decision Processes (MDPs) are a classical model for decision making in the presence of uncertainty. Often they are viewed as state transformers with planning objectives defined with respect to paths over MDP states. An increasingly popular alternative is to view them as distribution transformers, giving rise to a sequence of probability distributions over MDP states. For instance, reachability and safety properties in modeling robot swarms or chemical reaction networks are naturally defined in terms of probability distributions over states. Verifying such distributional properties is known to be hard and often beyond the reach of classical state-based verification techniques.\r\n\r\nIn this work, we consider the problems of certified policy (i.e. controller) verification and synthesis in MDPs under distributional reach-avoidance specifications. By certified we mean that, along with a policy, we also aim to synthesize a (checkable) certificate ensuring that the MDP indeed satisfies the property. Thus, given the target set of distributions and an unsafe set of distributions over MDP states, our goal is to either synthesize a certificate for a given policy or synthesize a policy along with a certificate, proving that the target distribution can be reached while avoiding unsafe distributions. To solve this problem, we introduce the novel notion of distributional reach-avoid certificates and present automated procedures for (1) synthesizing a certificate for a given policy, and (2) synthesizing a policy together with the certificate, both providing formal guarantees on certificate correctness. Our experimental evaluation demonstrates the ability of our method to solve several non-trivial examples, including a multi-agent robot-swarm model, to synthesize certified policies and to certify existing policies.","doi":"10.24963/ijcai.2024/1","url":"https://ijcai.org/proceedings/2024/1","pdf":"https://ijcai.org/proceedings/2024/0001.pdf","markdown":"https://ijcai.org/proceedings/2024/1.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":1,"pages":{"first":3,"last":12},"keywords":["Planning and Scheduling: PS: Markov decisions processes","Agent-based and Multi-agent Systems: MAS: Formal verification, validation and synthesis","Agent-based and Multi-agent Systems: MAS: Trust and reputation","Planning and Scheduling: PS: Planning under uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Agrawal et al., 2015] Manindra Agrawal, S. Akshay, Blaise Genest, and P. S. Thiagarajan. Approximate verification of the symbolic dynamics of markov chains. J. ACM, 62(1):2:1-2:34, 2015.","title":"Approximate verification of the symbolic dynamics of markov chains","authors":"S Manindra Agrawal, Blaise Akshay, P S Genest, Thiagarajan","year":"2015","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahmadi and Majumdar, 2016] Amir Ali Ahmadi and Anirudha Majumdar. Some applications of polynomial optimization in operations research and real-time decision making. Optim. Lett., 10(4):709-729, 2016.","title":"Some applications of polynomial optimization in operations research and real-time decision making","authors":"Majumdar ; Ahmadi, Ali Amir, Anirudha Ahmadi, Majumdar","year":"2016","venue":"Optim. Lett","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Akshay et al., 2015] S. Akshay, Timos Antonopoulos, Joël Ouaknine, and James Worrell. Reachability problems for markov chains. Inf. Process. Lett., 115(2):155-158, 2015.","title":"Reachability problems for markov chains","authors":"S Akshay, Timos Antonopoulos, Joël Ouaknine, James Worrell","year":"2015","venue":"Inf. Process. Lett","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Akshay et al., 2018] S. Akshay, Blaise Genest, and Nikhil Vyas. Distribution-based objectives for markov decision processes. In LICS, pages 36-45. ACM, 2018.","title":"Distribution-based objectives for markov decision processes","authors":"S Akshay, Blaise Genest, Nikhil Vyas","year":"2018","venue":"LICS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Akshay et al., 2023] S. Akshay, Krishnendu Chatterjee, To- bias Meggendorfer, and Dorde Zikelic. Mdps as distri- bution transformers: Affine invariant synthesis for safety objectives. In CAV (3), volume 13966 of Lecture Notes in Computer Science, pages 86-112. Springer, 2023.","title":"Tobias Meggendorfer, and Dorde Zikelic. Mdps as distribution transformers: Affine invariant synthesis for safety objectives","authors":"S Akshay, Krishnendu Chatterjee","year":"2023","venue":"CAV (3)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Akshay et al., 2024] S. Akshay, Krishnendu Chatterjee, To- bias Meggendorfer, and Ðor de Žikelić. Certified policy verification and synthesis for mdps under distributional reach-avoidance properties, 2024.","title":"Tobias Meggendorfer, and Ðor de Žikelić. Certified policy verification and synthesis for mdps under distributional reach-avoidance properties","authors":"S Akshay, Krishnendu Chatterjee","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Alshiekh et al., 2018] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In AAAI, pages 2669-2678. AAAI Press, 2018.","title":"Safe reinforcement learning via shielding","authors":"Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Asadi et al., 2021] Ali Asadi, Krishnendu Chatterjee, Hongfei Fu, Amir Kafshdar Goharshady, and Moham- mad Mahdavi. Polynomial reachability witnesses via stellensätze. In PLDI, pages 772-787. ACM, 2021.","title":"Polynomial reachability witnesses via stellensätze","authors":"Ali Asadi, Krishnendu Chatterjee, Hongfei Fu, Amir Kafshdar Goharshady, Mohammad Mahdavi","year":"2021","venue":"PLDI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Badings et al., 2022] Thom S. Badings, Alessandro Abate, Nils Jansen, David Parker, Hasan A. Poonawala, and Mar- iëlle Stoelinga. Sampling-based robust control of au- tonomous systems with non-gaussian noise. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 -March 1, 2022, pages 9669-9678. AAAI Press, 2022.","title":"Poonawala, and Mariëlle Stoelinga. Sampling-based robust control of autonomous systems with non-gaussian noise","authors":"Thom S Badings, Alessandro Abate, Nils Jansen, David Parker, A Hasan","year":"2022","venue":"Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Badings et al., 2023] Thom S. Badings, Licio Romao, Alessandro Abate, and Nils Jansen. Probabilities are not enough: Formal controller synthesis for stochastic dynam- ical models with epistemic uncertainty. In AAAI, pages 14701-14710. AAAI Press, 2023.","title":"Probabilities are not enough: Formal controller synthesis for stochastic dynamical models with epistemic uncertainty","authors":"Thom S Badings, Licio Romao, Alessandro Abate, Nils Jansen","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Ka- toen. Principles of model checking. MIT Press, 2008.","title":"Principles of model checking","authors":"Christel Baier, Joost-Pieter Katoen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Baldoni et al., 2008] Roberto Baldoni, François Bonnet, Alessia Milani, and Michel Raynal. On the solvability of anonymous partial grids exploration by mobile robots. In OPODIS, volume 5401 of Lecture Notes in Computer Science, pages 428-445. Springer, 2008.","title":"On the solvability of anonymous partial grids exploration by mobile robots","authors":"Roberto Baldoni, François Bonnet, Alessia Milani, Michel Raynal","year":"2008","venue":"OPODIS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Beauquier et al., 2006] Danièle Beauquier, Alexan- der Moshe Rabinovich, and Anatol Slissenko. A logic of probability with decidable model checking. J. Log. Comput., 16(4):461-487, 2006.","title":"A logic of probability with decidable model checking","authors":"Danièle Beauquier, Alexander Moshe Rabinovich, Anatol Slissenko","year":"2006","venue":"J. Log. Comput","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Cauchi and Abate, 2019] Nathalie Cauchi and Alessandro Abate. Stochy-automated verification and synthesis of stochastic processes. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, pages 258-259, 2019.","title":"Stochy-automated verification and synthesis of stochastic processes","authors":"Nathalie Cauchi, Alessandro Abate","year":"2019","venue":"Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Chadha et al., 2011] Rohit Chadha, Vijay Anand Kor- thikanti, Mahesh Viswanathan, Gul Agha, and YoungMin Kwon. Model checking mdps with a unique compact in- variant set of distributions. In Eighth International Confer- ence on Quantitative Evaluation of Systems, QEST 2011, Aachen, Germany, 5-8 September, 2011, pages 121-130. IEEE Computer Society, 2011.","title":"Model checking mdps with a unique compact invariant set of distributions","authors":"Rohit Chadha, Vijay Anand Korthikanti, Mahesh Viswanathan, Gul Agha, Youngmin Kwon","year":"2011","venue":"Eighth International Conference on Quantitative Evaluation of Systems, QEST 2011","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Cimatti et al., 2013] Alessandro Cimatti, Alberto Griggio, Bastiaan Schaafsma, and Roberto Sebastiani. The Math- SAT5 SMT Solver. In Nir Piterman and Scott Smolka, editors, Proceedings of TACAS, volume 7795 of LNCS. Springer, 2013.","title":"The Math-SAT5 SMT Solver","authors":"Alessandro Cimatti, Alberto Griggio, Bastiaan Schaafsma, Roberto Sebastiani","year":"2013","venue":"Proceedings of TACAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Colón and Sipma, 2001] Michael Colón and Henny Sipma. Synthesis of linear ranking functions. In TACAS, volume 2031 of Lecture Notes in Computer Science, pages 67-81. Springer, 2001.","title":"Synthesis of linear ranking functions","authors":"Sipma Colón, Michael Colón, Henny Sipma","year":"2001","venue":"TACAS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Colón et al., 2003] Michael Colón, Sriram Sankara- narayanan, and Henny Sipma. Linear invariant generation using non-linear constraint solving. In CAV, volume 2725 of Lecture Notes in Computer Science, pages 420-432. Springer, 2003.","title":"Linear invariant generation using non-linear constraint solving","authors":"Sriram Michael Colón, Henny Sankaranarayanan, Sipma","year":"2003","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[de Moura and Bjørner, 2008] Leonardo Mendonça de Moura and Nikolaj S. Bjørner. Z3: an efficient SMT solver. In TACAS, volume 4963 of Lecture Notes in Computer Science, pages 337-340. Springer, 2008.","title":"Z3: an efficient SMT solver","authors":"Leonardo Mendonça De Moura, Nikolaj S Bjørner","year":"2008","venue":"TACAS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Doyen et al., 2014] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Limit synchronization in markov decision processes. In FoSSaCS, volume 8412 of Lecture Notes in Computer Science, pages 58-72. Springer, 2014.","title":"Limit synchronization in markov decision processes","authors":"Laurent Doyen, Thierry Massart, Mahsa Shirmohammadi","year":"2014","venue":"FoSSaCS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Dutertre, 2014] Bruno Dutertre. Yices 2.2. In CAV, volume 8559 of Lecture Notes in Computer Science, pages 737-744. Springer, 2014.","title":"Bruno Dutertre","authors":"Dutertre","year":"2014","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Farkas, 1902] Julius Farkas. Theorie der einfachen ungle- ichungen. Journal für die reine und angewandte Mathe- matik (Crelles Journal), 1902(124):1-27, 1902.","title":"Theorie der einfachen ungleichungen","authors":"Farkas","year":"1902","venue":"Journal für die reine und angewandte Mathematik (Crelles Journal","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Gario and Micheli, 2015] Marco Gario and Andrea Micheli. Pysmt: a solver-agnostic library for fast prototyping of smt-based algorithms. In SMT workshop, volume 2015, 2015.","title":"Pysmt: a solver-agnostic library for fast prototyping of smt-based algorithms","authors":"Micheli Gario, Marco Gario, Andrea Micheli","year":"2015","venue":"SMT workshop","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Grover et al., 2022] Kush Grover, Jan Kretínský, Tobias Meggendorfer, and Maximilian Weininger. Anytime guar- antees for reachability in uncountable markov decision pro- cesses. In CONCUR, volume 243 of LIPIcs, pages 11:1- 11:20. Schloss Dagstuhl -Leibniz-Zentrum für Informatik, 2022.","title":"Anytime guarantees for reachability in uncountable markov decision processes","authors":"Grover","year":"2022","venue":"CONCUR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Gulwani et al., 2008] Sumit Gulwani, Saurabh Srivastava, and Ramarathnam Venkatesan. Program analysis as con- straint solving. In PLDI, pages 281-292. ACM, 2008.","title":"Program analysis as constraint solving","authors":"Sumit Gulwani, Saurabh Srivastava, Ramarathnam Venkatesan","year":"2008","venue":"PLDI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Henzinger et al., 2009] Thomas A. Henzinger, Maria Ma- teescu, and Verena Wolf. Sliding window abstraction for infinite markov chains. In CAV, volume 5643 of Lecture Notes in Computer Science, pages 337-352. Springer, 2009.","title":"Sliding window abstraction for infinite markov chains","authors":"Thomas A Henzinger, Maria Mateescu, Verena Wolf","year":"2009","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Jansen et al., 2020] Nils Jansen, Bettina Könighofer, Sebas- tian Junges, Alex Serban, and Roderick Bloem. Safe re- inforcement learning using probabilistic shields (invited paper). In CONCUR, volume 171 of LIPIcs, pages 3:1- 3:16. Schloss Dagstuhl -Leibniz-Zentrum für Informatik, 2020.","title":"Safe reinforcement learning using probabilistic shields","authors":"Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, Roderick Bloem","year":"2020","venue":"CONCUR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Jarvis-Wloszek et al., 2003] Zachary Jarvis-Wloszek, Ryan Feeley, Weehong Tan, Kunpeng Sun, and Andrew Packard. Some controls applications of sum of squares program- ming. In 42nd IEEE international conference on decision and control (IEEE Cat. No. 03CH37475), volume 5, pages 4676-4681. IEEE, 2003.","title":"Some controls applications of sum of squares programming","authors":"Zachary Jarvis-Wloszek, Ryan Feeley, Weehong Tan, Kunpeng Sun, Andrew Packard","year":"2003","venue":"42nd IEEE international conference on decision and control (IEEE Cat. No. 03CH37475","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Korthikanti et al., 2010] Vijay Anand Korthikanti, Mahesh Viswanathan, Gul Agha, and YoungMin Kwon. Reasoning about mdps as transformers of probability distributions. In QEST, pages 199-208. IEEE Computer Society, 2010.","title":"Reasoning about mdps as transformers of probability distributions","authors":"Anand Vijay, Mahesh Korthikanti, Gul Viswanathan, Youngmin Agha, Kwon","year":"2010","venue":"QEST","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Kwiatkowska et al., 2007] Marta Z. Kwiatkowska, Gethin Norman, and David Parker. Stochastic model checking. In SFM, volume 4486 of Lecture Notes in Computer Science, pages 220-270. Springer, 2007.","title":"Stochastic model checking","authors":"Marta Z Kwiatkowska, Gethin Norman, David Parker","year":"2007","venue":"SFM","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Kwon and Agha, 2011] YoungMin Kwon and Gul A. Agha. Verifying the evolution of probability distributions gov- erned by a DTMC. IEEE Trans. Software Eng., 37(1):126- 141, 2011.","title":"Verifying the evolution of probability distributions governed by a DTMC","authors":"Agha Kwon, Youngmin Kwon, Gul A Agha","year":"2011","venue":"IEEE Trans. Software Eng","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lavaei et al., 2020] Abolfazl Lavaei, Mahmoud Khaled, Sadegh Soudjani, and Majid Zamani. AMYTISS: par- allelized automated controller synthesis for large-scale stochastic systems. In CAV (2), volume 12225 of Lec- ture Notes in Computer Science, pages 461-474. Springer, 2020.","title":"AMYTISS: parallelized automated controller synthesis for large-scale stochastic systems","authors":"Abolfazl Lavaei, Mahmoud Khaled, Sadegh Soudjani, Majid Zamani","year":"2020","venue":"CAV (2)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Lechner et al., 2022] Mathias Lechner, Dorde Zikelic, Kr- ishnendu Chatterjee, and Thomas A. Henzinger. Stability verification in stochastic control systems via neural network supermartingales. In Thirty-Sixth AAAI Conference on Ar- tificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, Febru- ary 22 -March 1, 2022, pages 7326-7336. AAAI Press, 2022.","title":"Stability verification in stochastic control systems via neural network supermartingales","authors":"Mathias Lechner, Dorde Zikelic, Krishnendu Chatterjee, Thomas A Henzinger","year":"2022","venue":"Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Lipton et al., 2022] Richard Lipton, Florian Luca, Joris Nieuwveld, Joël Ouaknine, David Purser, and James Wor- rell. On the skolem problem and the skolem conjecture. In LICS, pages 5:1-5:9. ACM, 2022.","title":"On the skolem problem and the skolem conjecture","authors":"Richard Lipton, Florian Luca, Joris Nieuwveld, Joël Ouaknine, David Purser, James Worrell","year":"2022","venue":"LICS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Meurer et al., 2017] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondrej Certík, Sergey B. Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason Keith Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pe- dregosa, Matthew J. Curry, Andy R. Terrel, Stepán Roucka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony M. Scopatz. Sympy: symbolic computing in python. PeerJ Comput. Sci., 3:e103, 2017.","title":"Sympy: symbolic computing in python","authors":"Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondrej Certík, B Sergey, Matthew Kirpichev, Amit Rocklin, Sergiu Kumar, Jason Ivanov, Sartaj Keith Moore, Thilina Singh, Sean Rathnayake, Brian E Vig, Richard P Granger, Francesco Muller, Harsh Bonazzi, Shivam Gupta, Fredrik Vats, Fabian Johansson, Matthew J Pedregosa, Andy R Curry, Stepán Terrel, Ashutosh Roucka, Isuru Saboo, Sumith Fernando, Robert Kulal, Anthony M Cimrman, Scopatz","year":"2017","venue":"PeerJ Comput. Sci","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ouaknine and Worrell, 2012] Joël Ouaknine and James Wor- rell. Decision problems for linear recurrence sequences. In RP, volume 7550 of Lecture Notes in Computer Science, pages 21-28. Springer, 2012.","title":"Decision problems for linear recurrence sequences","authors":"Worrell Ouaknine, Joël Ouaknine, James Worrell","year":"2012","venue":"RP","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Prajna et al., 2007] Stephen Prajna, Ali Jadbabaie, and George J. Pappas. A framework for worst-case and stochas- tic safety verification using barrier certificates. IEEE Trans. Autom. Control., 52(8):1415-1428, 2007.","title":"A framework for worst-case and stochastic safety verification using barrier certificates","authors":"Stephen Prajna, Ali Jadbabaie, George J Pappas","year":"2007","venue":"IEEE Trans. Autom. Control","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":"Wiley Series in Probability and Statistics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Soudjani et al., 2015] Sadegh Esmaeil Zadeh Soudjani, Cas- par Gevaerts, and Alessandro Abate. FAUST 2 : Formal abstractions of uncountable-state stochastic processes. In TACAS, volume 9035 of Lecture Notes in Computer Sci- ence, pages 272-286. Springer, 2015.","title":"FAUST 2 : Formal abstractions of uncountable-state stochastic processes","authors":"Sadegh Esmaeil, Zadeh Soudjani, Caspar Gevaerts, Alessandro Abate","year":"2015","venue":"TACAS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Xue et al., 2021] Bai Xue, Renjue Li, Naijun Zhan, and Mar- tin Fränzle. Reach-avoid analysis for stochastic discrete- time systems. In ACC, pages 4879-4885. IEEE, 2021.","title":"Reach-avoid analysis for stochastic discretetime systems","authors":"Bai Xue, Renjue Li, Naijun Zhan, Martin Fränzle","year":"2021","venue":"ACC","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zikelic et al., 2023a] Dorde Zikelic, Mathias Lechner, Thomas A. Henzinger, and Krishnendu Chatterjee. Learn- ing control policies for stochastic systems with reach-avoid guarantees. In AAAI, pages 11926-11935. AAAI Press, 2023.","title":"Learning control policies for stochastic systems with reach-avoid guarantees","authors":"Zikelic","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zikelic et al., 2023b] Dorde Zikelic, Mathias Lechner, Abhi- nav Verma, Krishnendu Chatterjee, and Thomas A. Hen- zinger. Compositional policy learning in stochastic control systems with formal guarantees. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023.","title":"Compositional policy learning in stochastic control systems with formal guarantees","authors":"Zikelic","year":"2023","venue":"Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}