{"type":"ScholarlyArticle","title":"Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees","authors":[{"name":"Daqian Shao","slug":"daqian-shao","given":"Daqian","family":"Shao","affiliation":"Department of Computer Science , University of Oxford , UK"},{"name":"Marta Kwiatkowska","slug":"marta-kwiatkowska","given":"Marta","family":"Kwiatkowska","affiliation":"Department of Computer Science , University of Oxford , UK"}],"abstract":"Linear Temporal Logic (LTL) is widely used to specify high-level objectives for system policies, and it is highly desirable for autonomous systems to learn the optimal policy with respect to such specifications. However, learning the optimal policy from LTL specifications is not trivial. We present a model-free Reinforcement Learning (RL) approach that efficiently learns an optimal policy for an unknown stochastic system, modelled using Markov Decision Processes (MDPs). We propose a novel and more general product MDP, reward structure and discounting mechanism that, when applied in conjunction with off-the-shelf model-free RL algorithms, efficiently learn the optimal policy that maximizes the probability of satisfying a given LTL specification with optimality guarantees. We also provide improved theoretical results on choosing the key parameters in RL to ensure optimality. To directly evaluate the learned policy, we adopt probabilistic model checker PRISM to compute the probability of the policy satisfying such specifications. Several experiments on various tabular MDP environments across different LTL tasks demonstrate the improved sample efficiency and optimal policy convergence.","doi":"10.24963/ijcai.2023/465","url":"https://ijcai.org/proceedings/2023/465","pdf":"https://ijcai.org/proceedings/2023/0465.pdf","markdown":"https://ijcai.org/proceedings/2023/465.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/465","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":465,"pages":{"first":4180,"last":4189},"keywords":["Robotics: ROB: Learning in robotics","Machine Learning: ML: Reinforcement learning","Planning and Scheduling: PS: Markov decisions processes"],"videos":[],"references":[{"position":1,"raw":"[Aksaray et al., 2016] Derya Aksaray, Austin Jones, Zhao- dan Kong, Mac Schwager, and Calin Belta. Q-learning for robust satisfaction of signal temporal logic specifica- tions. 2016 IEEE 55th Conference on Decision and Con- trol, CDC 2016, pages 6565-6570, 9 2016.","title":"Q-learning for robust satisfaction of signal temporal logic specifications","authors":"Derya Aksaray, Austin Jones, Zhaodan Kong, Mac Schwager, Calin Belta","year":"2016","venue":"IEEE 55th Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alur et al., 2021] Rajeev Alur, Suguman Bansal, Osbert Bastani, and Kishor Jothimurugan. A framework for transforming specifications in reinforcement learning. arXiv:2111.00272, 2021.","title":"A framework for transforming specifications in reinforcement learning","authors":"Rajeev Alur, Suguman Bansal, Osbert Bastani, Kishor Jothimurugan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andreas et al., 2016] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learn- ing with policy sketches. 34th International Conference on Machine Learning, ICML 2017, 1:229-239, 11 2016.","title":"Modular multitask reinforcement learning with policy sketches","authors":"Jacob Andreas, Dan Klein, Sergey Levine","year":"2016","venue":"34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Katoen. Principles Of Model Checking, volume 950. MIT Press, 2008.","title":"Principles Of Model Checking","authors":"Christel Baier, Joost-Pieter Katoen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bozkurt et al., 2019] Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, and Miroslav Pajic. Control synthe- sis from linear temporal logic specifications using model- free reinforcement learning. Proceedings -IEEE Inter- national Conference on Robotics and Automation, pages 10349-10355, 2019.","title":"Control synthesis from linear temporal logic specifications using modelfree reinforcement learning","authors":"Kamil Alper, Yu Bozkurt, Michael M Wang, Miroslav Zavlanos, Pajic","year":"2019","venue":"Proceedings -IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba Openai. OpenAI Gym. arXiv:1606.01540, 6 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba Openai","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cai et al., 2021] Mingyu Cai, Mohammadhosein Hasan- beig, Shaoping Xiao, Alessandro Abate, and Zhen Kan. Modular deep reinforcement learning for continuous mo- tion planning with temporal logic. IEEE Robotics and Au- tomation Letters, 6(4):7973-7980, 2021.","title":"Modular deep reinforcement learning for continuous motion planning with temporal logic","authors":"Mingyu Cai, Mohammadhosein Hasanbeig, Shaoping Xiao, Alessandro Abate, Zhen Kan","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Camacho et al., 2019] Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. LTL and beyond: Formal languages for reward function specification in reinforce- ment learning. IJCAI International Joint Conference on Artificial Intelligence, 2019-August:6065-6073, 2019.","title":"LTL and beyond: Formal languages for reward function specification in reinforcement learning","authors":"Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, Sheila A Mcilraith","year":"2019","venue":"IJCAI International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/840"},{"position":9,"raw":"[De Giacomo and Vardi, 2013] Giuseppe De Giacomo and Moshe Y Vardi. Linear temporal logic and linear dynamic logic on finite traces. Proceedings of the AAAI Conference on Artificial Intelligence, 2013.","title":"Linear temporal logic and linear dynamic logic on finite traces","authors":"De Giacomo, Giuseppe Vardi, Giacomo De, Moshe Y Vardi","year":"2013","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/132"},{"position":10,"raw":"[Devlin and Kudenko, 2012] Sam Devlin and Daniel Ku- denko. Dynamic potential-based reward shaping. Proceed- ings of the 11th International Conference on Autonomous Agents and Multiagent Systems, 2012.","title":"Dynamic potential-based reward shaping","authors":"Kudenko Devlin, Sam Devlin, Daniel Kudenko","year":"2012","venue":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Giuseppe De Giacomo et al., 2019] Giuseppe De Giacomo, Luca Iocchi, Marco Favorito, and Fabio Patrizi. Foun- dations for restraining bolts: reinforcement learning with LTLf/LDLf restraining specifications. In Proceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling, 2019.","title":"Foundations for restraining bolts: reinforcement learning with LTLf/LDLf restraining specifications","authors":"Giuseppe De, Giacomo","year":"2019","venue":"Proceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hahn et al., 2019] Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Omega-regular objectives in model-free rein- forcement learning. Proceedings of the International Con- ference on Tools and Algorithms for the Construction and Analysis of Systems, 11427 LNCS:395-412, 2019.","title":"Omega-regular objectives in model-free reinforcement learning","authors":"Hahn","year":"2019","venue":"Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hahn et al., 2020] Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Faithful and effective reward schemes for model-free reinforcement learning of omega-regular ob- jectives. Automated Technology for Verification and Anal- ysis, 12302 LNCS:108-124, 2020.","title":"Faithful and effective reward schemes for model-free reinforcement learning of omega-regular objectives","authors":"Hahn","year":"2020","venue":"Automated Technology for Verification and Analysis","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hasanbeig et al., 2019] M. Hasanbeig, Y. Kantaros, A. Abate, D. Kroening, G. J. Pappas, and I. Lee. Rein- forcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees. Proceed- ings of the IEEE Conference on Decision and Control, 2019-December:5338-5343, 9 2019.","title":"Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees","authors":"M Hasanbeig, Y Kantaros, A Abate, D Kroening, G J Pappas, I Lee","year":"2019","venue":"Proceedings of the IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hasanbeig et al., 2020] Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. Deep rein- forcement learning with temporal logics. Proceedings of the International Conference on Formal Modeling and Analysis of Timed Systems, 12288 LNCS:1-22, 2020.","title":"Deep reinforcement learning with temporal logics","authors":"Hasanbeig","year":"2020","venue":"Proceedings of the International Conference on Formal Modeling and Analysis of Timed Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jiang et al., 2021] Yuqian Jiang, Suda Bharadwaj, Bo Wu, Rishi Shah, Ufuk Topcu, and Peter Stone. Temporal-logic- based reward shaping for continuing reinforcement learn- ing tasks. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 35(9):7995-8003, 5 2021.","title":"Temporal-logicbased reward shaping for continuing reinforcement learning tasks","authors":"Jiang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jothimurugan et al., 2020] Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. A composable specification lan- guage for reinforcement learning tasks. Advances in Neu- ral Information Processing Systems, 32, 8 2020.","title":"A composable specification language for reinforcement learning tasks","authors":"Kishor Jothimurugan, Rajeev Alur, Osbert Bastani","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jothimurugan et al., 2021] Kishor Jothimurugan, Suguman Bansal, Osbert Bastani, and Rajeev Alur. Compo- sitional reinforcement learning from logical specifica- tions. Advances in Neural Information Processing Sys- tems, 12:10026-10039, 6 2021.","title":"Compositional reinforcement learning from logical specifications","authors":"Kishor Jothimurugan, Suguman Bansal, Osbert Bastani, Rajeev Alur","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Křetínský et al., 2018] Jan Křetínský, Tobias Meggendor- fer, Salomon Sickert, and Christopher Ziegler. Rabinizer 4: From LTL to your favourite deterministic automaton. Proceedings of the International Conference on Computer Aided Verification, 10981 LNCS:567-577, 2018.","title":"Rabinizer 4: From LTL to your favourite deterministic automaton","authors":"Křetínský","year":"2018","venue":"Proceedings of the International Conference on Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kwiatkowska et al., 2011] Marta Kwiatkowska, Gethin Norman, and David Parker. PRISM 4.0: Verification of probabilistic real-time systems. Computer Aided Verification, 6806 LNCS:585-591, 2011.","title":"PRISM 4.0: Verification of probabilistic real-time systems","authors":"Marta Kwiatkowska, Gethin Norman, David Parker","year":"2011","venue":"Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2016] Xiao Li, Cristian Ioan Vasile, and Calin Belta. Reinforcement learning with temporal logic re- wards. IEEE International Conference on Intelligent Robots and Systems, 2017-September:3834-3839, 12 2016.","title":"Reinforcement learning with temporal logic rewards","authors":"Xiao Li, Cristian Ioan Vasile, Calin Belta","year":"2016","venue":"IEEE International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Littman et al., 2017] Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Isbell, Min Wen, and James MacGlashan. Environment-independent task specifications via GLTL. arXiv:1704.04341, 4 2017.","title":"Environment-independent task specifications via GLTL","authors":"L Michael, Ufuk Littman, Jie Topcu, Charles Fu, Min Isbell, James Wen, Macglashan","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Littman, 2001] M.L. Littman. Markov decision processes. International Encyclopedia of the Social & Behavioral Sciences, pages 9240-9242, 2001.","title":"Markov decision processes","authors":"M L Littman","year":"2001","venue":"International Encyclopedia of the Social & Behavioral Sciences","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ng et al., 1999] Andrew Y. Ng, Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under re- ward transformations: Theory and application to reward shaping. Proceedings of the Sixteenth International Con- ference on Machine Learning, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Y Ng, Andrew Y Ng, Daishi Harada, Stuart Russell","year":"1999","venue":"Proceedings of the Sixteenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sadigh et al., 2014] Dorsa Sadigh, Eric S. Kim, Samuel Coogan, S. Shankar Sastry, and Sanjit A. Seshia. A learn- ing based approach to control synthesis of Markov de- cision processes for linear temporal logic specifications. Proceedings of the IEEE Conference on Decision and Control, 2015-February(February):1091-1096, 2014.","title":"A learning based approach to control synthesis of Markov decision processes for linear temporal logic specifications","authors":"Dorsa Sadigh, Eric S Kim, S Shankar Samuel Coogan, Sanjit A Sastry, Seshia","year":"2014","venue":"Proceedings of the IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shao and Kwiatkowska, 2023] Daqian Shao and Marta Kwiatkowska. Sample efficient model-free reinforcement learning from ltl specifications with optimality guarantees. arXiv:2305.01381, 5 2023.","title":"Sample efficient model-free reinforcement learning from ltl specifications with optimality guarantees","authors":"Kwiatkowska Shao, Daqian Shao, Marta Kwiatkowska","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sickert et al., 2016] Salomon Sickert, Javier Esparza, Ste- fan Jaax, and Jan Křetínský. Limit-deterministic Büchi automata for linear temporal logic. Computer Aided Veri- fication, 9780:312-332, 2016.","title":"Limit-deterministic Büchi automata for linear temporal logic","authors":"Sickert","year":"2016","venue":"Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An Introduction, vol- ume 3. MIT Press, 2018.","title":"Reinforcement learning: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Toro Icarte et al., 2018] Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. Us- ing reward machines for high-level task specification and decomposition in reinforcement learning. Proceedings of the 35th International Conference on Machine Learning, 2018.","title":"Using reward machines for high-level task specification and decomposition in reinforcement learning","authors":"Rodrigo Toro Icarte, Richard Toryn Q Klassen, Sheila A Valenzano, Mcllraith","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Toro Icarte et al., 2022] Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. Re- ward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73:173-208, 2022.","title":"Reward machines: Exploiting reward function structure in reinforcement learning","authors":"Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, Sheila A Mcilraith","year":"2022","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Watkins and Dayan, 1992] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning 1992 8:3, 8(3):279-292, 5 1992.","title":"Q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu and Topcu, 2019] Zhe Xu and Ufuk Topcu. Transfer of temporal logic formulas in reinforcement learning. Pro- ceedings of the Twenty-Eighth International Joint Confer- ence on Artificial Intelligence, 2019.","title":"Transfer of temporal logic formulas in reinforcement learning","authors":"Zhe Xu, Ufuk Topcu","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/557"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}