{"type":"ScholarlyArticle","title":"Steady-State Policy Synthesis for Verifiable Control","authors":[{"name":"Alvaro Velasquez","slug":"alvaro-velasquez","given":"Alvaro","family":"Velasquez","affiliation":"Information Directorate , Air Force Research Laboratory"}],"abstract":"In this paper, we introduce the Steady-State Policy Synthesis (SSPS) problem which consists of finding a stochastic decision-making policy that maximizes expected rewards while satisfying a set of asymptotic behavioral specifications. These specifications are determined by the steady-state probability distribution resulting from the Markov chain induced by a given policy. Since such distributions necessitate recurrence, we propose a solution which finds policies that induce recurrent Markov chains within possibly non-recurrent Markov Decision Processes (MDPs). The SSPS problem functions as a generalization of steady-state control, which has been shown to be in PSPACE. We improve upon this result by showing that SSPS is in P via linear programming. Our results are validated using CPLEX simulations on MDPs with over 10000 states. We also prove that the deterministic variant of SSPS is NP-hard.","doi":"10.24963/ijcai.2019/784","url":"https://ijcai.org/proceedings/2019/784","pdf":"https://ijcai.org/proceedings/2019/0784.pdf","markdown":"https://ijcai.org/proceedings/2019/784.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/784","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Planning and Scheduling","track":"Main track","number":784,"pages":{"first":5653,"last":5661},"keywords":["Planning and Scheduling: Model-Based Reasoning","Uncertainty in AI: Markov Decision Processes","Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning under Uncertainty","Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis"],"videos":[],"references":[{"position":1,"raw":"[Akshay et al., 2013] Sundararaman Akshay, Nathalie Bertrand, Serge Haddad, and Loic Helouet. The steady- state control problem for markov decision processes. In International Conference on Quantitative Evaluation of Systems, pages 290-304. Springer, 2013.","title":"The steadystate control problem for markov decision processes","authors":"Sundararaman Akshay, Nathalie Bertrand, Serge Haddad, Loic Helouet","year":"2013","venue":"International Conference on Quantitative Evaluation of Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baumgartner et al., 2018] Peter Baumgartner, Sylvie Thiébaux, and Felipe Trevizan. Heuristic search planning with multi-objective probabilistic ltl constraints. 2018.","title":"Heuristic search planning with multi-objective probabilistic ltl constraints","authors":"Peter Baumgartner, Sylvie Thiébaux, Felipe Trevizan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Canny, 1988] John Canny. Some algebraic and geometric computations in pspace. In Proceedings of the twentieth annual ACM symposium on Theory of computing, pages 460-467. ACM, 1988.","title":"Some algebraic and geometric computations in pspace","authors":"John Canny, Canny","year":"1988","venue":"Proceedings of the twentieth annual ACM symposium on Theory of computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cinlar, 2013] Erhan Cinlar. Introduction to stochastic pro- cesses. Courier Corporation, 2013.","title":"Introduction to stochastic processes","authors":"Erhan Cinlar","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[CPL, 2017] Ibm ilog cplex optimization studio v12.8.0 doc- umentation. IBM Knowledge Center, 2017.","title":"Ibm ilog cplex optimization studio v12","authors":"Cpl","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feyzabadi, 2017] Seyedshams Feyzabadi. Robot Planning with Constrained Markov Decision Processes. PhD thesis, UNIVERSITY OF CALIFORNIA, MERCED, 2017.","title":"Seyedshams Feyzabadi. Robot Planning with Constrained Markov Decision Processes","authors":"Feyzabadi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Floudas and Visweswaran, 1995] Christodoulos A Floudas and V Visweswaran. Quadratic optimization. In Handbook of global optimization, pages 217-269. Springer, 1995.","title":"Christodoulos A Floudas and V Visweswaran","authors":"Visweswaran Floudas","year":"1995","venue":"Handbook of global optimization","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karp, 1972] Richard M Karp. Reducibility among combi- natorial problems. In Complexity of computer computa- tions, pages 85-103. Springer, 1972.","title":"Reducibility among combinatorial problems","authors":"M Richard, Karp","year":"1972","venue":"Complexity of computer computations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kearns and Singh, 2002] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in poly- nomial time. Machine learning, 49(2-3):209-232, 2002.","title":"Near-optimal reinforcement learning in polynomial time","authors":"Michael Kearns, Satinder Singh","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konstantopoulos, 2009] Takis Konstantopoulos. Markov chains and random walks. Lecture notes, 2009.","title":"Markov chains and random walks","authors":"Takis Konstantopoulos","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[McCuaig, 1993] William McCuaig. Intercyclic digraphs. Contemporary Mathematics, 147:203-203, 1993.","title":"Intercyclic digraphs","authors":"William Mccuaig, Mccuaig","year":"1993","venue":"Contemporary Mathematics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ross, 2014] Sheldon M Ross. Introduction to probability models. Academic press, 2014.","title":"Introduction to probability models","authors":"M Sheldon, Ross","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Russell et al., 2015] Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial ar- tificial intelligence. Ai Magazine, 36(4):105-114, 2015.","title":"Research priorities for robust and beneficial artificial intelligence","authors":"Stuart Russell, Daniel Dewey, Max Tegmark","year":"2015","venue":"Ai Magazine","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Teichteil-Königsbuch, 2012] Florent Teichteil-Königsbuch. Path-constrained markov decision processes: bridging the gap between probabilistic model-checking and decision- theoretic planning. In 20th European Conference on Arti- ficial Intelligence (ECAI 2012), 2012.","title":"Path-constrained markov decision processes: bridging the gap between probabilistic model-checking and decisiontheoretic planning","authors":"Florent Teichteil-Königsbuch","year":"2012","venue":"20th European Conference on Artificial Intelligence (ECAI 2012)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Trevizan et al., 2016] Felipe W Trevizan, Sylvie Thiébaux, Pedro Henrique Santana, and Brian Charles Williams. Heuristic search in dual space for constrained stochastic shortest path problems. In ICAPS, pages 326-334, 2016.","title":"Heuristic search in dual space for constrained stochastic shortest path problems","authors":"Sylvie Felipe W Trevizan, Pedro Henrique Thiébaux, Brian Charles Santana, Williams","year":"2016","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tsitsiklis, 2007] John N Tsitsiklis. Np-hardness of checking the unichain condition in average cost mdps. Operations research letters, 35(3):319-323, 2007.","title":"Np-hardness of checking the unichain condition in average cost mdps","authors":"John N Tsitsiklis","year":"2007","venue":"Operations research letters","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wolff and Murray, 2016] Eric M Wolff and Richard M Mur- ray. Optimal control of nonlinear systems with temporal logic specifications. In Robotics Research, pages 21-37. Springer, 2016.","title":"Optimal control of nonlinear systems with temporal logic specifications","authors":"M Eric, Richard M Wolff, Murray","year":"2016","venue":"Robotics Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}