{"type":"ScholarlyArticle","title":"Steady-State Policy Synthesis in Multichain Markov Decision Processes","authors":[{"name":"George Atia","slug":"george-atia","given":"George","family":"Atia","affiliation":"Department of Electrical and Computer Engineering , University of Central Florida"},{"name":"Andre Beckus","slug":"andre-beckus","given":"Andre","family":"Beckus","affiliation":"Department of Electrical and Computer Engineering , University of Central Florida"},{"name":"Ismail Alkhouri","slug":"ismail-alkhouri","given":"Ismail","family":"Alkhouri","affiliation":"Department of Electrical and Computer Engineering , University of Central Florida"},{"name":"Alvaro Velasquez","slug":"alvaro-velasquez","given":"Alvaro","family":"Velasquez","affiliation":"Information Directorate , Air Force Research Laboratory"}],"abstract":"The formal synthesis of automated or autonomous agents has elicited strong interest from the artificial intelligence community in recent years. This problem space broadly entails the derivation of decision-making policies for agents acting in an environment such that a formal specification of behavior is satisfied. Popular formalisms for such specifications include the quintessential Linear Temporal Logic (LTL) and Computation Tree Logic (CTL) which reason over infinite sequences and trees, respectively, of states. However, the related and relevant problem of reasoning over the frequency with which states are visited infinitely and enforcing behavioral specifications on the same has received little attention. That problem, known as Steady-State Policy Synthesis (SSPS) or steady-state control, is the focus of this paper. Prior related work has been mostly confined to unichain Markov Decision Processes (MDPs), while a tractable solution to the general multichain setting heretofore remains elusive. In this paper, we provide a solution to the latter within the context of multichain MDPs over a class of policies that account for all possible transitions in the given MDP. The solution policy is derived from a novel linear program (LP) that encodes constraints on the limiting distributions of the Markov chain induced by said policy. We establish a one-to-one correspondence between the feasible solutions of the LP and the stationary distributions of the induced Markov chains. The derived policy is shown to maximize the reward among the constrained class of stationary policies and to satisfy the specification constraints even when it does not exercise all possible transitions.","doi":"10.24963/ijcai.2020/563","url":"https://ijcai.org/proceedings/2020/563","pdf":"https://ijcai.org/proceedings/2020/0563.pdf","markdown":"https://ijcai.org/proceedings/2020/563.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/563","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Planning and Scheduling","track":"Main track","number":563,"pages":{"first":4069,"last":4075},"keywords":["Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning under Uncertainty","Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24862/file_24862.m3u8","length_sec":293,"provider":"hls"}],"references":[{"position":1,"raw":"[Akshay et al., 2013] Sundararaman Akshay, Nathalie Bertrand, Serge Haddad, and Loïc Hélouët. The steady- state control problem for Markov decision processes. In Int. Conf. Quant. Eval., pages 290-304, Berlin Heidelberg, 2013. Springer.","title":"The steadystate control problem for Markov decision processes","authors":"Sundararaman Akshay, Nathalie Bertrand, Serge Haddad, Loïc Hélouët","year":"2013","venue":"Int. Conf. Quant. Eval","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Altman et al., 2019] Eitan Altman, Said Boularouk, and Di- dier Josselin. Constrained Markov decision processes with total expected cost criteria. In Proceedings of the 12th EAI International Conference on Performance Evaluation Methodologies and Tools, pages 191-192. ACM, 2019.","title":"Constrained Markov decision processes with total expected cost criteria","authors":"Eitan Altman, Said Boularouk, Didier Josselin","year":"2019","venue":"Proceedings of the 12th EAI International Conference on Performance Evaluation Methodologies and Tools","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision processes. CRC Press, Boca Raton, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bhatnagar and Lakshmanan, 2012] Shalabh Bhatnagar and K Lakshmanan. An online actor-critic algorithm with function approximation for constrained Markov decision processes. Journal of Optimization Theory and Applica- tions, 153(3):688-708, 2012.","title":"An online actor-critic algorithm with function approximation for constrained Markov decision processes","authors":"Shalabh Bhatnagar, Lakshmanan","year":"2012","venue":"Journal of Optimization Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boussemart and Limnios, 2004] M Boussemart and N Limnios. Markov decision processes with asymptotic average failure rate constraint. Communications in Statistics-Theory and Methods, 33(7):1689-1714, 2004.","title":"Markov decision processes with asymptotic average failure rate constraint","authors":"M Boussemart, Limnios","year":"2004","venue":"Communications in Statistics-Theory and Methods","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Boussemart et al., 2002] M Boussemart, N Limnios, and JC Fillion. Non-ergodic Markov decision processes with a constraint on the asymptotic failure rate: general class of policies. Stochastic models, 18(1):173-191, 2002.","title":"Non-ergodic Markov decision processes with a constraint on the asymptotic failure rate: general class of policies","authors":"Boussemart, J C Limnios, Fillion","year":"2002","venue":"Stochastic models","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[De Ghellinck, 1960] Guy De Ghellinck. Les problèmes de décisions séquentielles. Cahiers du Centre d'Etudes de Recherche Opérationnelle, 2(2):161-179, 1960.","title":"Les problèmes de décisions séquentielles","authors":"Guy De, Ghellinck","year":"1960","venue":"Cahiers du Centre d'Etudes de Recherche Opérationnelle","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Derman, 1970] Cyrus Derman. Finite State Markovian De- cision Processes. Academic Press, Inc., Orlando, FL, USA, 1970.","title":"Cyrus Derman. Finite State Markovian Decision Processes","authors":"Derman","year":"1970","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Feinberg, 2009] E. A. Feinberg. Adaptive computation of optimal nonrandomized policies in constrained average- reward MDPs. In IEEE Symposium on Adaptive Dy- namic Programming and Reinforcement Learning, pages 96-100, March 2009.","title":"Adaptive computation of optimal nonrandomized policies in constrained averagereward MDPs","authors":"E A Feinberg","year":"2009","venue":"IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feller, 1968] William Feller. An Introduction to Probability Theory and its Applications, volume 1. Wiley, 3 edition, 1968.","title":"An Introduction to Probability Theory and its Applications","authors":"William Feller","year":"1968","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fisher et al., 2013] Michael Fisher, Louise A Dennis, and Matthew P Webster. Verifying autonomous systems. Com- mun. ACM, 56(9):84-93, 2013.","title":"Verifying autonomous systems","authors":"Michael Fisher, Louise A Dennis, Matthew P Webster","year":"2013","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Haesaert et al., 2015] Sofie Haesaert, Alessandro Abate, and Paul MJ Van den Hof. Correct-by-design output feed- back of LTI systems. In 54th IEEE Conference on Deci- sion and Control (CDC), pages 6159-6164, 2015.","title":"Correct-by-design output feedback of LTI systems","authors":"Sofie Haesaert, Alessandro Abate, Paul Mj Van Den Hof","year":"2015","venue":"54th IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kallenberg, 1983] L. C. M. Kallenberg. Linear program- ming and finite Markovian control problems. Mathema- tisch Centrum, Amsterdam, 1983.","title":"Linear programming and finite Markovian control problems","authors":"L C M Kallenberg","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kress-Gazit et al., 2018] Hadas Kress-Gazit, Morteza Lahi- janian, and Vasumathi Raman. Synthesis for robots: Guar- antees and feedback for robot behavior. Annual Review of Control, Robotics, and Autonomous Systems, 1:211-236, 2018.","title":"Synthesis for robots: Guarantees and feedback for robot behavior","authors":"Kress-Gazit","year":"2018","venue":"Annual Review of Control, Robotics, and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lazar, 1983] Andreas Lazar. Optimal flow control of a class of queueing networks in equilibrium. IEEE Transactions on Automatic Control, 28(11):1001-1007, 1983.","title":"Optimal flow control of a class of queueing networks in equilibrium","authors":"Andreas Lazar","year":"1983","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Manne, 1960] Alan S. Manne. Linear programming and se- quential decisions. Management Science, 6(3):259-267, 1960.","title":"Linear programming and sequential decisions","authors":"Alan S Manne","year":"1960","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Puterman, 1994] Martin Puterman. Markov decision pro- cesses : discrete stochastic dynamic programming. Wiley, New York, 1994.","title":"Markov decision processes : discrete stochastic dynamic programming","authors":"Martin Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ross, 1989] Keith W. Ross. Randomized and past- dependent policies for Markov decision processes with multiple constraints. Operations Research, 37(3):474- 477, 1989.","title":"Randomized and pastdependent policies for Markov decision processes with multiple constraints","authors":"Keith W Ross","year":"1989","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Skwirzynski, 1981] Joseph K Skwirzynski. New concepts in multi-user communication, volume 43. Springer Science & Business Media, 1981.","title":"New concepts in multi-user communication","authors":"K Joseph, Skwirzynski","year":"1981","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tarjan, 1971] R. Tarjan. Depth-first search and linear graph algorithms. In 12th Annual Symposium on Switching and Automata Theory, pages 114-121, Oct 1971.","title":"Depth-first search and linear graph algorithms","authors":"R Tarjan","year":"1971","venue":"12th Annual Symposium on Switching and Automata Theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Velasquez, 2019] Alvaro Velasquez. Steady-state policy synthesis for verifiable control. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5653-5661. AAAI Press, 2019.","title":"Steady-state policy synthesis for verifiable control","authors":"Alvaro Velasquez","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/784"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}