{"type":"ScholarlyArticle","title":"Computational Approaches for Stochastic Shortest Path on Succinct MDPs","authors":[{"name":"Krishnendu Chatterjee","slug":"krishnendu-chatterjee","given":"Krishnendu","family":"Chatterjee","affiliation":"IST Austria"},{"name":"Hongfei Fu","slug":"hongfei-fu","given":"Hongfei","family":"Fu","affiliation":"Shanghai Jiao Tong University"},{"name":"Amir Goharshady","slug":"amir-goharshady","given":"Amir","family":"Goharshady","affiliation":"IST Austria"},{"name":"Nastaran Okati","slug":"nastaran-okati","given":"Nastaran","family":"Okati","affiliation":"Ferdowsi University of Mashhad"}],"abstract":"We consider the stochastic shortest path (SSP) problem for succinct Markov decision processes (MDPs), where the MDP consists of a set of variables, and a set of nondeterministic rules that update the variables. First, we show that several examples from the AI literature can be modeled as succinct MDPs. Then we present computational approaches for upper and lower bounds for the SSP problem: (a) for computing upper bounds, our method is polynomial-time in the implicit description of the MDP; (b) for lower bounds, we present a polynomial-time (in the size of the implicit description) reduction to quadratic programming. Our approach is applicable even to infinite-state MDPs. Finally, we present experimental results to demonstrate the effectiveness of our approach on several classical examples from the AI literature.","doi":"10.24963/ijcai.2018/653","url":"https://ijcai.org/proceedings/2018/653","pdf":"https://ijcai.org/proceedings/2018/0653.pdf","markdown":"https://ijcai.org/proceedings/2018/653.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/653","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":653,"pages":{"first":4700,"last":4707},"keywords":["Planning and Scheduling: Planning under Uncertainty","Planning and Scheduling: Theoretical Foundations of Planning","Planning and Scheduling: Markov Decisions Processes"],"videos":[],"references":[{"position":1,"raw":"[Berkelaar et al., 2004] Michel Berkelaar, Kjell Eikland, Pe- ter Notebaert, et al. LPsolve: Open source (mixed-integer) linear programming system. Technical report, 2004.","title":"LPsolve: Open source (mixed-integer) linear programming system","authors":"Michel Berkelaar, Kjell Eikland, Peter Notebaert","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 2005] Dimitri Bertsekas. Dynamic programming and optimal control. Athena Scientific, 3rd edition, 2005.","title":"Dynamic programming and optimal control","authors":"Dimitri Bertsekas","year":"2005","venue":"Athena Scientific","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boutilier and Lu, 2016] Craig Boutilier and Tyler Lu. Bud- get allocation using weakly coupled, constrained Markov decision processes. In UAI, pages 52-61, 2016.","title":"Budget allocation using weakly coupled, constrained Markov decision processes","authors":"Craig Boutilier, Tyler Lu","year":"2016","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chakarov and Sankaranarayanan, 2013] Aleksandar Chakarov and Sriram Sankaranarayanan. Probabilis- tic program analysis with martingales. In CAV, pages 511-526, 2013.","title":"Aleksandar Chakarov and Sriram Sankaranarayanan. Probabilistic program analysis with martingales","authors":"Sankaranarayanan Chakarov","year":"2013","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chatterjee et al., 2016] Krishnendu Chatterjee, Hongfei Fu, Petr Novotnỳ, and Rouzbeh Hasheminezhad. Algorithmic analysis of qualitative and quantitative termination prob- lems for affine probabilistic programs. In POPL, pages 327-342, 2016.","title":"Algorithmic analysis of qualitative and quantitative termination problems for affine probabilistic programs","authors":"Krishnendu Chatterjee, Hongfei Fu, Petr Novotnỳ, Rouzbeh Hasheminezhad","year":"2016","venue":"POPL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chatterjee et al., 2018] Krishnendu Chatterjee, Hongfei Fu, Amir Goharshady, and Nastaran Okati. Computational ap- proaches for stochastic shortest path on succinct MDPs. arXiv preprint arXiv:1804.08984, 2018.","title":"Computational approaches for stochastic shortest path on succinct MDPs","authors":"Krishnendu Chatterjee, Hongfei Fu, Amir Goharshady, Nastaran Okati","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dean et al., 1997] Thomas Dean, Robert Givan, and Sonia Leach. Model reduction techniques for computing approx- imately optimal solutions for Markov decision processes. In UAI, pages 124-131, 1997.","title":"Model reduction techniques for computing approximately optimal solutions for Markov decision processes","authors":"Thomas Dean, Robert Givan, Sonia Leach","year":"1997","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Delgado et al., 2011] Karina Valdivia Delgado, Scott San- ner, and Leliane Nunes De Barros. Efficient solutions to factored MDPs with imprecise transition probabilities. Ar- tif. Intell., 175(9-10):1498-1527, 2011.","title":"Efficient solutions to factored MDPs with imprecise transition probabilities","authors":"Karina Valdivia Delgado, Scott Sanner, Leliane Nunes, De Barros","year":"2011","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Farkas, 1894] Julius Farkas. A Fourier-Féle mechanikai elv alkalmazásai (Hungarian). Mathematikaiés Ter- mészettudományi Értesitö, 12:457-472, 1894.","title":"A Fourier-Féle mechanikai elv alkalmazásai (Hungarian)","authors":"Julius Farkas","year":"1894","venue":"Mathematikaiés Természettudományi Értesitö","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ferns et al., 2004] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite Markov decision pro- cesses. In UAI, pages 162-169, 2004.","title":"Metrics for finite Markov decision processes","authors":"Ferns","year":"2004","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Filar and Vrieze, 1997] Jerzy Filar and Koos Vrieze. Com- petitive Markov Decision Processes. Springer, 1997.","title":"Competitive Markov Decision Processes","authors":"Vrieze Filar, Jerzy Filar, Koos Vrieze","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gilbert et al., 2017] Hugo Gilbert, Paul Weng, and Yan Xu. Optimizing quantiles in preference-based Markov decision processes. In AAAI, pages 3569-3575, 2017.","title":"Optimizing quantiles in preference-based Markov decision processes","authors":"Gilbert","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guestrin et al., 2003] Carlos Guestrin, Daphne Koller, Ronald Parr, and Shobha Venkataraman. Efficient solution algorithms for factored MDPs. J. Artif. Intell. Res., 19:399-468, 2003.","title":"Efficient solution algorithms for factored MDPs","authors":"Carlos Guestrin, Daphne Koller, Ronald Parr, Shobha Venkataraman","year":"2003","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hansen and Abdoulahi, 2015] Eric A Hansen and Ibrahim Abdoulahi. Efficient bounds in heuristic search algorithms for stochastic shortest path problems. In AAAI, pages 3283-3290, 2015.","title":"Efficient bounds in heuristic search algorithms for stochastic shortest path problems","authors":"Eric A Hansen, Ibrahim Abdoulahi","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Howard, 1960] Ronald A Howard. Dynamic Programming and Markov Processes. MIT Press, 1960.","title":"Dynamic Programming and Markov Processes","authors":"A Ronald, Howard","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lukasiewycz, 2008] Martin Lukasiewycz. JavaILP -java interface to ILP solvers, javailp.sourceforge.net, 2008.","title":"JavaILP -java interface to ILP solvers","authors":"Martin Lukasiewycz","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Motzkin, 1936] Theodore Samuel Motzkin. Beiträge zur Theorie der linearen Ungleichungen (German). PhD the- sis, Basel, Jerusalem, 1936.","title":"Theodore Samuel Motzkin. Beiträge zur Theorie der linearen Ungleichungen","authors":"Motzkin","year":"1936","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Perrault and Boutilier, 2017] Andrew Perrault and Craig Boutilier. Multiple-profile prediction-of-use games. In AAMAS, pages 275-295, 2017.","title":"Multiple-profile prediction-of-use games","authors":"Boutilier Perrault, Andrew Perrault, Craig Boutilier","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":"/proceedings/2017/52"},{"position":19,"raw":"[Poupart et al., 2015] Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, and Michael Bowl- ing. Approximate linear programming for constrained par- tially observable Markov decision processes. In AAAI, pages 3342-3348, 2015.","title":"Approximate linear programming for constrained partially observable Markov decision processes","authors":"Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, Michael Bowling","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Puterman, 1994] Martin L Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. Wi- ley, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sanner, 2010] Scott Sanner. Relational dynamic influence diagram language (RDDL): Language description. Tech- nical report, 2010.","title":"Relational dynamic influence diagram language (RDDL): Language description","authors":"Scott Sanner","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sigaud and Buffet, 2010] Olivier Sigaud and Olivier Buffet. Markov Decision Processes in Artificial Intelligence. Wi- ley, 2010.","title":"Markov Decision Processes in Artificial Intelligence","authors":"Buffet Sigaud, Olivier Sigaud, Olivier Buffet","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Singh et al., 1994] Satinder P Singh, Tommi Jaakkola, and Michael I Jordan. Learning without state-estimation in partially observable Markovian decision processes. In Ma- chine Learning Proceedings, pages 284 -292.","title":"Learning without state-estimation in partially observable Markovian decision processes","authors":"P Satinder, Tommi Singh, Michael I Jaakkola, Jordan","year":null,"venue":"Machine Learning Proceedings","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Morgan Kaufmann, 1994.","title":null,"authors":"Morgan Kaufmann","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tivellato, 2017] Alberto Tivellato. JOptimizer -Java convex optimizer, www.joptimizer.com, 2017.","title":"JOptimizer -Java convex optimizer","authors":"Alberto Tivellato","year":"2017","venue":null,"doi":null,"url":"www.joptimizer.com","internal_url":null},{"position":26,"raw":"[Topin et al., 2015] Nicholay Topin, Nicholas Haltmeyer, Shawn Squire, John Winder, Marie desJardins, and James MacGlashan. Portable option discovery for automated learning transfer in object-oriented Markov decision pro- cesses. In IJCAI, pages 3856-3864, 2015.","title":"Portable option discovery for automated learning transfer in object-oriented Markov decision processes","authors":"Topin","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/542"},{"position":27,"raw":"[Williams and Young, 2007] Jason D Williams and Steve Young. Partially observable Markov decision processes for spoken dialog systems. Comput. Speech Lang., 21(2):393- 422, 2007.","title":"Partially observable Markov decision processes for spoken dialog systems","authors":"Jason D Williams, Steve Young","year":"2007","venue":"Comput. Speech Lang","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Williams, 1991] David Williams. Probability with Martin- gales. Cambridge University Press, 1991.","title":"Probability with Martingales","authors":"David Williams","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Younes and Littman, 2004] Håkan Younes and Michael L Littman. PPDDL1. 0: The language for the probabilistic part of IPC-4. In IPC, pages 70-74, 2004.","title":"PPDDL1. 0: The language for the probabilistic part of IPC-4","authors":"Littman Younes, Håkan Younes, Michael L Littman","year":"2004","venue":"IPC","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}