{"type":"ScholarlyArticle","title":"Improved Strong Worst-case Upper Bounds for MDP Planning","authors":[{"name":"Anchit Gupta","slug":"anchit-gupta","given":"Anchit","family":"Gupta","affiliation":"Department of Computer Science and Engineering , Indian Institute of Technology Bombay"},{"name":"Shivaram Kalyanakrishnan","slug":"shivaram-kalyanakrishnan","given":"Shivaram","family":"Kalyanakrishnan","affiliation":"Department of Computer Science and Engineering , Indian Institute of Technology Bombay"}],"abstract":"The Markov Decision Problem (MDP) plays a central role in AI as an abstraction of sequential decision making. We contribute to the theoretical analysis of MDP PLANNING, which is the problem of computing an optimal policy for a given MDP. Specifically, we furnish improved STRONG WORST-CASE upper bounds on the running time of MDP planning. Strong bounds are those that depend only on the number of states n and the number of actions k in the specified MDP; they have no dependence on affiliated variables such as the discount factor and the number of bits needed to represent the MDP. Worst-case bounds apply to EVERY run of an algorithm; randomised algorithms can typically yield faster EXPECTED running times. While the special case of 2-action MDPs (that is, k = 2) has recently received some attention, bounds for general k have remained to be improved for several decades. Our contributions are to this general case. For k >= 3, the tightest strong upper bound shown to date for MDP planning belongs to a family of algorithms called Policy Iteration. This bound is only a polynomial improvement over a trivial bound of poly(n, k) k^{n} [Mansour and Singh, 1999]. In this paper, we generalise a contrasting algorithm called the Fibonacci Seesaw, and derive a bound of poly(n, k) k^{0.6834n}. The key construct we use is a template to map algorithms for the 2-action setting to the general setting. Interestingly, this idea can also be used to design Policy Iteration algorithms with a running time upper bound of poly(n, k) k^{0.7207n}. Both our results improve upon bounds that have stood for several decades.","doi":"10.24963/ijcai.2017/248","url":"https://ijcai.org/proceedings/2017/248","pdf":"https://ijcai.org/proceedings/2017/0248.pdf","markdown":"https://ijcai.org/proceedings/2017/248.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/248","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":248,"pages":{"first":1788,"last":1794},"keywords":["Uncertainty in AI: Sequential Decision Making","Planning and Scheduling: Markov Decisions Processes","Machine Learning: Reinforcement Learning","Planning and Scheduling: Planning Algorithms"],"videos":[],"references":[{"position":1,"raw":"[Bellman, 1957] Richard Bellman. Dynamic Programming. Princeton University Press, 1 st edition, 1957.","title":"Dynamic Programming","authors":"Richard Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 2012] Dimitri P. Bertsekas. Dynamic Program- ming and Optimal Control, volume 2. Athena Scientific, 4 th edition, 2012.","title":"Dynamic Programming and Optimal Control","authors":"Dimitri P Bertsekas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chazelle and Matousek, 1996] Bernard Chazelle and Jirí Matousek. On linear-time deterministic algorithms for op- timization problems in fixed dimension. Journal of Algo- rithms, 21(3):579-597, 1996.","title":"On linear-time deterministic algorithms for optimization problems in fixed dimension","authors":"Matousek Chazelle, Bernard Chazelle, Jirí Matousek","year":"1996","venue":"Journal of Algorithms","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fearnley, 2010] John Fearnley. Exponential lower bounds for policy iteration. In Proceedings of the Thirty-seventh International Colloquium on Automata, Languages and Programming (ICALP 2010), pages 551-562. Springer, 2010.","title":"Exponential lower bounds for policy iteration","authors":"John Fearnley","year":"2010","venue":"Proceedings of the Thirty-seventh International Colloquium on Automata, Languages and Programming (ICALP 2010)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gärtner, 2002] Bernd Gärtner. The random-facet simplex algorithm on combinatorial cubes. Random Structures and Algorithms, 20(3):353-381, 2002.","title":"The random-facet simplex algorithm on combinatorial cubes","authors":"Bernd Gärtner","year":"2002","venue":"Random Structures and Algorithms","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hansen and Zwick, 2010] Thomas Dueholm Hansen and Uri Zwick. Lower bounds for Howard's algorithm for find- ing minimum mean-cost cycles. In Proceedings of the Twenty-second International Symposium on Algorithms and Computation (ISAAC 2011), pages 425-426. Springer, 2010.","title":"Lower bounds for Howard's algorithm for finding minimum mean-cost cycles","authors":"Thomas Dueholm, Hansen, Uri Zwick","year":"2010","venue":"Proceedings of the Twenty-second International Symposium on Algorithms and Computation (ISAAC 2011)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hollanders et al., 2012] Romain Hollanders, Balázs Gerencsér, and Jean-Charles Delvenne. The complexity of policy iteration is exponential for discounted Markov decision processes. In Proceedings of the Fifty-first IEEE Conference on Decision and Control (CDC 2012), pages 5997-6002. IEEE, 2012.","title":"The complexity of policy iteration is exponential for discounted Markov decision processes","authors":"Romain Hollanders, Balázs Gerencsér, Jean-Charles Delvenne","year":"2012","venue":"Proceedings of the Fifty-first IEEE Conference on Decision and Control (CDC 2012)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hollanders et al., 2014] Romain Hollanders, Balázs Gerencsér, Jean-Charles Delvenne, and Raphaël M. Jungers. Improved bound on the worst case complexity of policy iteration, 2014. URL: http://arxiv.org/pdf/1410.7583v1.pdf.","title":"Improved bound on the worst case complexity of policy iteration","authors":"Romain Hollanders, Balázs Gerencsér, Jean-Charles Delvenne, Raphaël M Jungers","year":"2014","venue":null,"doi":null,"url":"http://arxiv.org/pdf/1410.7583v1.pdf","internal_url":null},{"position":9,"raw":"[Howard, 1960] Ronald A. Howard. Dynamic programming and Markov processes. MIT Press, 1960.","title":"Dynamic programming and Markov processes","authors":"Ronald A Howard","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaggi, 2006] Martin Jaggi. Linear and quadratic programming by unique sink orientations, 2006. Diploma thesis in Mathematics, ETH Zürich. URL: http://www.m8j.net/data/List/Files-106/DA.pdf.","title":"Linear and quadratic programming by unique sink orientations","authors":"Martin Jaggi","year":"2006","venue":null,"doi":null,"url":"http://www.m8j.net/data/List/Files-106/DA.pdf","internal_url":null},{"position":11,"raw":"[Kalyanakrishnan et al., 2016a] Shivaram Kalyanakrishnan, Utkarsh Mall, and Ritish Goyal. Batch-switching policy iteration. In Proceedings of the Twenty-fifth International Joint Conference on Artificial Intelligence (IJCAI 2016), pages 3147-3153. AAAI Press, 2016.","title":"Batch-switching policy iteration","authors":"Kalyanakrishnan","year":"2016","venue":"Proceedings of the Twenty-fifth International Joint Conference on Artificial Intelligence (IJCAI 2016)","doi":null,"url":null,"internal_url":"/proceedings/2016/446"},{"position":12,"raw":"[Kalyanakrishnan et al., 2016b] Shivaram Kalyanakrishnan, Neeldhara Misra, and Aditya Gopalan. Randomised pro- cedures for initialising and switching actions in policy it- eration. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI 2016), pages 3145-3151. AAAI Press, 2016.","title":"Randomised procedures for initialising and switching actions in policy iteration","authors":"Kalyanakrishnan","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI 2016)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Littman et al., 1995] Michael L. Littman, Thomas L. Dean, and Leslie Pack Kaelbling. On the complexity of solv- ing Markov decision problems. In Proceedings of the Eleventh Annual Conference on Uncertainty in Artificial Intelligence (UAI 1995), pages 394-402.","title":"On the complexity of solving Markov decision problems","authors":"L Michael, Thomas L Littman, Leslie Dean, Kaelbling Pack","year":null,"venue":"Proceedings of the Eleventh Annual Conference on Uncertainty in Artificial Intelligence (UAI 1995)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Morgan Kauf- mann, 1995.","title":null,"authors":"Morgan Kaufmann","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Madani et al., 2010] Omid Madani, Mikkel Thorup, and Uri Zwick. Discounted deterministic Markov decision pro- cesses and discounted all-pairs shortest paths. ACM Trans- actions on Algorithms, 6(2):33:1-33:25, 2010.","title":"Discounted deterministic Markov decision processes and discounted all-pairs shortest paths","authors":"Omid Madani, Mikkel Thorup, Uri Zwick","year":"2010","venue":"ACM Transactions on Algorithms","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mansour and Singh, 1999] Yishay Mansour and Satinder Singh. On the complexity of policy iteration. In Pro- ceedings of the Fifteenth Conference on Uncertainty in Ar- tificial Intelligence (UAI 1999), pages 401-408.","title":"On the complexity of policy iteration","authors":"Yishay Mansour, Satinder Singh","year":null,"venue":"Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence (UAI 1999)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Morgan Kaufmann, 1999.","title":null,"authors":"Morgan Kaufmann","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Matoušek et al., 1996] Jiří Matoušek, Micha Sharir, and Emo Welzl. A subexponential bound for linear program- ming. Algorithmica, 16(4/5):498-516, 1996.","title":"A subexponential bound for linear programming","authors":"Matoušek","year":"1996","venue":"Algorithmica","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Megiddo, 1984] Nimrod Megiddo. Linear programming in linear time when the dimension is fixed. Journal of the ACM, 31(1):114-127, 1984.","title":"Nimrod Megiddo. Linear programming in linear time when the dimension is fixed","authors":"Megiddo","year":"1984","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Melekopoglou and Condon, 1994] Mary Melekopoglou and Anne Condon. On the complexity of the policy improvement algorithm for Markov decision processes. INFORMS Journal on Computing, 6(2):188-192, 1994.","title":"Mary Melekopoglou and Anne Condon. On the complexity of the policy improvement algorithm for Markov decision processes","authors":"Condon Melekopoglou","year":"1994","venue":"INFORMS Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Post and Ye, 2013] Ian Post and Yinyu Ye. The simplex method is strongly polynomial for deterministic Markov decision processes. In Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2013), pages 1465-1473.","title":"Ian Post and Yinyu Ye. The simplex method is strongly polynomial for deterministic Markov decision processes","authors":"Ye Post","year":"2013","venue":"Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2013)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Morgan Kaufmann, 2013.","title":null,"authors":"Morgan Kaufmann","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes. Wiley, 1994.","title":"Markov Decision Processes","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rüst, 2007] Leonard Yves Rüst. The P -Matrix Com- plementarity Problem: Generalizations and specializa- tions, 2007. Dissertation for the degree of Doctor of Sciences, Swiss Federal Institute of Technology. URL: https://www.inf.ethz.ch/personal/emo/DoctThesisFiles/ ruest07.pdf.","title":"The P -Matrix Complementarity Problem: Generalizations and specializations","authors":"Leonard Yves, Rüst","year":"2007","venue":null,"doi":null,"url":"https://www.inf.ethz.ch/personal/emo/DoctThesisFiles/ruest07.pdf","internal_url":null},{"position":25,"raw":"[Scheder, 2011] Dominik Alban Scheder. Algorithms and Extremal Properties of SAT and CSP. PhD thesis, Swiss Federal Institute of Technology Zurich, 2011. URL: http://basics.sjtu.edu.cn/ dominik/publications/ Dissertation-Dominik-Scheder.pdf.","title":"Algorithms and Extremal Properties of SAT and CSP","authors":"Alban Dominik, Scheder","year":"2011","venue":null,"doi":null,"url":"http://basics.sjtu.edu.cn/dominik/publications/Dissertation-Dominik-Scheder.pdf","internal_url":null},{"position":26,"raw":"[Szabó and Welzl, 2001] Tibor Szabó and Emo Welzl. Unique sink orientations of cubes. In Proceedings of the Forty-second Annual Symposium on Foundations of Com- puter Science (FOCS 2001), pages 547-555. IEEE, 2001.","title":"Unique sink orientations of cubes","authors":"Welzl Szabó, Tibor Szabó, Emo Welzl","year":"2001","venue":"Proceedings of the Forty-second Annual Symposium on Foundations of Computer Science (FOCS 2001)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}