{"type":"ScholarlyArticle","title":"Reduction Techniques for Model Checking and Learning in MDPs","authors":[{"name":"Suda Bharadwaj","slug":"suda-bharadwaj","given":"Suda","family":"Bharadwaj","affiliation":"* University of Texas at Austin"},{"name":"Stephane Le Roux","slug":"stephane-le-roux","given":"Stephane","family":"Le Roux","affiliation":"† Université libre de Bruxelles"},{"name":"Guillermo Perez","slug":"guillermo-perez","given":"Guillermo","family":"Perez","affiliation":"† Université libre de Bruxelles"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"* University of Texas at Austin"}],"abstract":"Omega-regular objectives in Markov decision processes (MDPs) reduce to reachability: find a policy which maximizes the probability of reaching a target set of states. Given an MDP, an initial distribution, and a target set of states, such a policy can be computed by most probabilistic model checking tools. If the MDP is only partially specified, i.e., some prob- abilities are unknown, then model-learning techniques can be used to statistically approximate the probabilities and enable the computation of the de- sired policy. For fully specified MDPs, reducing the size of the MDP translates into faster model checking; for partially specified MDPs, into faster learning. We provide reduction techniques that al- low us to remove irrelevant transition probabilities: transition probabilities (known, or to be learned) that do not influence the maximal reachability probability. Among other applications, these reductions can be seen as a pre-processing of MDPs before model checking or as a way to reduce the number of experiments required to obtain a good approximation of an unknown MDP.","doi":"10.24963/ijcai.2017/597","url":"https://ijcai.org/proceedings/2017/597","pdf":"https://ijcai.org/proceedings/2017/0597.pdf","markdown":"https://ijcai.org/proceedings/2017/597.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/597","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Planning and Scheduling","track":"Main track","number":597,"pages":{"first":4273,"last":4279},"keywords":["Multidisciplinary Topics and Applications: Validation and Verification","Uncertainty in AI: Sequential Decision Making","Planning and Scheduling: Markov Decisions Processes","Uncertainty in AI: Uncertainty in AI"],"videos":[],"references":[{"position":1,"raw":"[Aspnes and Herlihy, 1990] James Aspnes and Maurice Herlihy. Fast randomized consensus using shared memory. J. of Algo- rithms, 11(3):441-461, September 1990.","title":"Fast randomized consensus using shared memory","authors":"Herlihy Aspnes, James Aspnes, Maurice Herlihy","year":"1990","venue":"J. of Algorithms","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Katoen. Principles of model checking. MIT Press, 2008.","title":"Principles of model checking","authors":"Christel Baier, Joost-Pieter Katoen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bernstein et al., 2002] Daniel S. Bernstein, Robert Givan, Neil Im- merman, and Shlomo Zilberstein. The complexity of decentral- ized control of Markov decision processes. Math. Oper. Res., 27(4):819-840, November 2002.","title":"The complexity of decentralized control of Markov decision processes","authors":"S Daniel, Robert Bernstein, Neil Givan, Shlomo Immerman, Zilberstein","year":"2002","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brafman and Tennenholtz, 2003] Ronen I. Brafman and Moshe Tennenholtz. R-max -a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res., 3:213- 231, March 2003.","title":"R-max -a general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2003","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brázdil et al., 2014] Tomás Brázdil, Krishnendu Chatterjee, Martin Chmelik, Vojtech Forejt, Jan Kretínský, Marta Z. Kwiatkowska, David Parker, and Mateusz Ujma. Verification of Markov decision processes using learning algorithms. In ATVA, volume 8837 of LNCS, pages 98-114, 2014.","title":"Verification of Markov decision processes using learning algorithms","authors":"Tomás Brázdil, Krishnendu Chatterjee, Martin Chmelik, Vojtech Forejt, Jan Kretínský, Marta Z Kwiatkowska, David Parker, Mateusz Ujma","year":"2014","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ciesinski et al., 2008] Frank Ciesinski, Christel Baier, Marcus Größer, and Joachim Klein. Reduction techniques for model checking Markov decision processes. In QEST, pages 45-54, 2008.","title":"Reduction techniques for model checking Markov decision processes","authors":"Frank Ciesinski, Christel Baier, Marcus Größer, Joachim Klein","year":"2008","venue":"QEST","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Condon, 1992] Anne Condon. The complexity of stochastic games. Inf. Comput., 96(2):203-224, 1992.","title":"The complexity of stochastic games","authors":"Anne Condon","year":"1992","venue":"Inf. Comput","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Courcoubetis and Yannakakis, 1995] Costas Courcoubetis and Mi- halis Yannakakis. The complexity of probabilistic verification. J. ACM, 42(4):857-907, 1995.","title":"The complexity of probabilistic verification","authors":"Costas Courcoubetis, Mihalis Yannakakis","year":"1995","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[De Alfaro, 1997] Luca De Alfaro. Formal verification of proba- bilistic systems. PhD thesis, Stanford University, 1997.","title":"Formal verification of probabilistic systems","authors":"Luca De, Alfaro","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dehnert et al., 2017] Christian Dehnert, Sebastian Junges, Joost- Pieter Katoen, and Matthias Volk. A storm is coming: A modern probabilistic model checker. In CAV, 2017.","title":"A storm is coming: A modern probabilistic model checker","authors":"Dehnert","year":"2017","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ding et al., 2014] X. Ding, S. L. Smith, C. Belta, and D. Rus. Op- timal control of Markov decision processes with linear temporal logic constraints. IEEE Trans. on Automatic Control, 59(5):1244- 1257, May 2014.","title":"Optimal control of Markov decision processes with linear temporal logic constraints","authors":"Ding","year":"2014","venue":"IEEE Trans. on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fu and Topcu, 2014] Jie Fu and Ufuk Topcu. Probably approxi- mately correct MDP learning and control with temporal logic constraints. In RSS, 2014.","title":"Probably approximately correct MDP learning and control with temporal logic constraints","authors":"Topcu ; ] Jie Fu, Ufuk Fu, Topcu","year":"2014","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Haddad and Monmege, 2014] Serge Haddad and Benjamin Mon- mege. Reachability in mdps: Refining convergence of value itera- tion. In Joël Ouaknine, Igor Potapov, and James Worrell, editors, RP, volume 8762 of LNCS, pages 125-137. Springer, 2014.","title":"Reachability in mdps: Refining convergence of value iteration","authors":"Serge Haddad, Benjamin Monmege","year":"2014","venue":"LNCS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaelbling et al., 1996] Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: A survey. JAIR, 4:237-285, 1996.","title":"Reinforcement learning: A survey","authors":"Leslie Pack Kaelbling, Michael L Littman, Andrew W Moore","year":"1996","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kawaguchi, 2016] Kenji Kawaguchi. Bounded optimal exploration in MDP. In AAAI, pages 1758-1764, 2016.","title":"Bounded optimal exploration in MDP","authors":"Kenji Kawaguchi","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kearns and Singh, 2002] Michael Kearns and Satinder Singh. Near- optimal reinforcement learning in polynomial time. Machine Learning, 49(2):209-232, 2002.","title":"Nearoptimal reinforcement learning in polynomial time","authors":"Michael Kearns, Satinder Singh","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kolobov et al., 2011] Andrey Kolobov, Mausam, Daniel S. Weld, and Hector Geffner. Heuristic search for generalized stochastic shortest path mdps. In Fahiem Bacchus, Carmel Domshlak, Stefan Edelkamp, and Malte Helmert, editors, ICAPS. AAAI, 2011.","title":"Heuristic search for generalized stochastic shortest path mdps","authors":"Andrey Kolobov, Daniel S Mausam, Hector Weld, Geffner","year":"2011","venue":"ICAPS. AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kolter and Ng, 2009] J. Z. Kolter and Andrew Y. Ng. Near- bayesian exploration in polynomial time. In Andrea P. Danyluk, Lon Bottou, and Michael L. Littman, editors, ICML, page 65, 2009.","title":"Nearbayesian exploration in polynomial time","authors":"Kolter, ; ] J Z Ng, Andrew Y Kolter, Ng","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kwiatkowska et al., 2001] Marta Z. Kwiatkowska, Gethin Norman, and Roberto Segala. Automated verification of a randomized distributed consensus protocol using Cadence SMV and PRISM. In CAV, pages 194-206, 2001.","title":"Automated verification of a randomized distributed consensus protocol using Cadence SMV and PRISM","authors":"Marta Z Kwiatkowska, Gethin Norman, Roberto Segala","year":"2001","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kwiatkowska et al., 2004] Marta Z. Kwiatkowska, Gethin Norman, David Parker, and Jeremy Sproston. Performance analysis of probabilistic timed automata using digital clocks. In FORMATS, pages 105-120, 2004.","title":"Performance analysis of probabilistic timed automata using digital clocks","authors":"Marta Z Kwiatkowska, Gethin Norman, David Parker, Jeremy Sproston","year":"2004","venue":"FORMATS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kwiatkowska et al., 2011] Marta Z. Kwiatkowska, Gethin Norman, and David Parker. PRISM 4.0: Verification of probabilistic real- time systems. In Ganesh Gopalakrishnan and Shaz Qadeer, editors, CAV, volume 6806 of LNCS, pages 585-591. Springer, 2011.","title":"PRISM 4.0: Verification of probabilistic realtime systems","authors":"Marta Z Kwiatkowska, Gethin Norman, David Parker","year":"2011","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lacerda et al., 2015] Bruno Lacerda, David Parker, and Nick Hawes. Optimal policy generation for partially satisfiable co- safe LTL specifications. In IJCAI, pages 1587-1593, 2015.","title":"Optimal policy generation for partially satisfiable cosafe LTL specifications","authors":"Lacerda","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lahijanian et al., 2010] M. Lahijanian, J. Wasniewski, S. B. Ander- sson, and C. Belta. Motion planning and control from temporal logic specifications with probabilistic satisfaction guarantees. In ICRA, pages 3227-3232, 2010.","title":"Motion planning and control from temporal logic specifications with probabilistic satisfaction guarantees","authors":"M Lahijanian, J Wasniewski, S B Andersson, C Belta","year":"2010","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Littman, 1996] Michael L. Littman. Algorithms for Sequential De- cision Making. PhD thesis, Brown University, 1996.","title":"Algorithms for Sequential Decision Making","authors":"L Michael, Littman","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos H. Papadimitriou and John N. Tsitsiklis. The complexity of Markov Decision Processes. Math. Oper. Res., 12:441-450, 1987.","title":"The complexity of Markov Decision Processes","authors":"Christos H Papadimitriou, John N Tsitsiklis","year":"1987","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Puterman, 2005] Martin L. Puterman. Markov Decision Processes. Wiley-Interscience, 2005.","title":"Markov Decision Processes","authors":"Martin L Puterman","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Russell and Norvig, 2010] Stuart J. Russell and Peter Norvig. Arti- ficial Intelligence -A Modern Approach (3. internat. ed.). Pearson Education, 2010.","title":"Artificial Intelligence -A Modern Approach","authors":"Stuart J Russell, Peter Norvig","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Russell et al., 2015] Stuart J. Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. AI Magazine, 36(4), 2015.","title":"Research priorities for robust and beneficial artificial intelligence","authors":"Stuart J Russell, Daniel Dewey, Max Tegmark","year":"2015","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Steinmetz et al., 2016] Marcel Steinmetz, Jörg Hoffmann, and Olivier Buffet. Goal probability analysis in probabilistic planning: Exploring and enhancing the state of the art. JAIR, 57:229-271, 2016.","title":"Goal probability analysis in probabilistic planning: Exploring and enhancing the state of the art","authors":"Steinmetz","year":"2016","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Strehl et al., 2009] Alexander L. Strehl, Lihong Li, and Michael L. Littman. Reinforcement learning in finite MDPs: PAC analysis. J. Mach. Learn. Res., 10:2413-2444, 2009.","title":"Reinforcement learning in finite MDPs: PAC analysis","authors":"Alexander L Strehl, Lihong Li, Michael L Littman","year":"2009","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Svorenová et al., 2013] María Svorenová, Ivana Cerna, and Calin Belta. Optimal control of MDPs with temporal logic constraints. In CDC, pages 3938-3943, 2013.","title":"Optimal control of MDPs with temporal logic constraints","authors":"Svorenová","year":"2013","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Valiant, 2013] Leslie Valiant. Probably Approximately Correct: Nature's Algorithms for Learning and Prospering in a Complex World. Basic Books, 2013.","title":"Leslie Valiant. Probably Approximately Correct: Nature's Algorithms for Learning and Prospering in a Complex World","authors":"Valiant","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wen and Topcu, 2016] Min Wen and Ufuk Topcu. Probably ap- proximately correct learning in stochastic games with temporal logic specifications. In Subbarao Kambhampati, editor, IJCAI, pages 3630-3636, 2016.","title":"Probably approximately correct learning in stochastic games with temporal logic specifications","authors":"Wen, Topcu ;, Min Wen, Ufuk Topcu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/511"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}