{"type":"ScholarlyArticle","title":"On the (In)Tractability of Reinforcement Learning for LTL Objectives","authors":[{"name":"Cambridge Yang","slug":"cambridge-yang","given":"Cambridge","family":"Yang","affiliation":"MIT CSAIL"},{"name":"Michael L. Littman","slug":"michael-l-littman","given":"Michael L.","family":"Littman","affiliation":"Brown University"},{"name":"Michael Carbin","slug":"michael-carbin","given":"Michael","family":"Carbin","affiliation":"MIT CSAIL"}],"abstract":"In recent years, researchers have made significant progress in devising reinforcement-learning algorithms for optimizing linear temporal logic (LTL) objectives and LTL-like objectives.\r\nDespite these advancements, there are fundamental limitations to how well this problem can be solved. Previous studies have alluded to this fact but have not examined it in depth.\r\nIn this paper, we address the tractability of reinforcement learning for general LTL objectives from a theoretical perspective.\r\nWe formalize the problem under the probably approximately correct learning in Markov decision processes (PAC-MDP) framework, a standard framework for measuring sample complexity in reinforcement learning.\r\nIn this formalization, we prove that the optimal policy for any LTL formula is PAC-MDP-learnable if and only if the formula is in the most limited class in the LTL hierarchy, consisting of formulas that are decidable within a finite horizon.\r\nPractically, our result implies that it is impossible for a reinforcement-learning algorithm to obtain a PAC-MDP guarantee on the performance of its learned policy after finitely many interactions with an unconstrained environment for LTL objectives that are not decidable within a finite horizon.","doi":"10.24963/ijcai.2022/507","url":"https://ijcai.org/proceedings/2022/507","pdf":"https://ijcai.org/proceedings/2022/0507.pdf","markdown":"https://ijcai.org/proceedings/2022/507.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/507","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":507,"pages":{"first":3650,"last":3658},"keywords":["Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987072","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985215","length_sec":659,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alur et al., 2021] Rajeev Alur, Suguman Bansal, Osbert Bastani, and Kishor Jothimurugan. A framework for trans- forming specifications in reinforcement learning. arXiv preprint: 2111.00272, 2021.","title":"A framework for transforming specifications in reinforcement learning","authors":"Rajeev Alur, Suguman Bansal, Osbert Bastani, Kishor Jothimurugan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ashok et al., 2019] Pranav Ashok, Jan Křetínský, and Max- imilian Weininger. Pac statistical model checking for markov decision processes and stochastic games. In CAV, 2019.","title":"Pac statistical model checking for markov decision processes and stochastic games","authors":"Pranav Ashok, Jan Křetínský, Maximilian Weininger","year":"2019","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bozkurt et al., 2020] Alper Bozkurt, Yu Wang, Michael Za- vlanos, and Miroslav Pajic. Control synthesis from linear temporal logic specifications using model-free reinforce- ment learning. In ICRA, 2020.","title":"Control synthesis from linear temporal logic specifications using model-free reinforcement learning","authors":"Alper Bozkurt, Yu Wang, Michael Zavlanos, Miroslav Pajic","year":"2020","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brázdil et al., 2014] Tomáš Brázdil, Krishnendu Chatterjee, Martin Chmelík, Vojtěch Forejt, Jan Křetínský, Marta Kwiatkowska, David Parker, and Mateusz Ujma. Verifi- cation of markov decision processes using learning algo- rithms. In ATVA, 2014.","title":"Verification of markov decision processes using learning algorithms","authors":"Brázdil","year":"2014","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Camacho et al., 2019] Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. Ltl and beyond: Formal lan- guages for reward function specification in reinforcement learning. In IJCAI, 2019.","title":"Ltl and beyond: Formal languages for reward function specification in reinforcement learning","authors":"Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, Sheila A Mcilraith","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/840"},{"position":6,"raw":"[Dann et al., 2019] Christoph Dann, Lihong Li, Wei Wei, and Emma Brunskill. Policy certificates: Towards ac- countable reinforcement learning. In ICML, 2019.","title":"Policy certificates: Towards accountable reinforcement learning","authors":"Christoph Dann, Lihong Li, Wei Wei, Emma Brunskill","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dewey, 2014] Dan Dewey. Reinforcement learning and the reward engineering principle. In AAAI Spring Symposia, 2014.","title":"Reinforcement learning and the reward engineering principle","authors":"Dan Dewey, Dewey","year":"2014","venue":"AAAI Spring Symposia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fiechter, 1994] Claude-Nicolas Fiechter. Efficient rein- forcement learning. In COLT, 1994.","title":"Efficient reinforcement learning","authors":"Claude-Nicolas Fiechter","year":"1994","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fu and Topcu, 2014] Jie Fu and Ufuk Topcu. Probably ap- proximately correct MDP learning and control with tem- poral logic constraints. In Robotics: Science and Systems X, 2014.","title":"Probably approximately correct MDP learning and control with temporal logic constraints","authors":"Topcu ; ] Jie Fu, Ufuk Fu, Topcu","year":"2014","venue":"Robotics: Science and Systems X","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Giacomo et al., 2019] Giuseppe De Giacomo, L. Iocchi, Marco Favorito, and F. Patrizi. Foundations for restrain- ing bolts: Reinforcement learning with ltlf/ldlf restraining specifications. In ICAPS, 2019.","title":"Foundations for restraining bolts: Reinforcement learning with ltlf/ldlf restraining specifications","authors":"Giuseppe De, Giacomo, L Iocchi, Marco Favorito, F Patrizi","year":"2019","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grill et al., 2016] Jean-Bastien Grill, Michal Valko, and R. Munos. Blazing the trails before beating the path: Sample-efficient monte-carlo planning. In NIPS, 2016.","title":"Blazing the trails before beating the path: Sample-efficient monte-carlo planning","authors":"Jean-Bastien Grill, Michal Valko, R Munos","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hahn et al., 2019] Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Omega-regular objectives in model-free rein- forcement learning. In TACAS, 2019.","title":"Omega-regular objectives in model-free reinforcement learning","authors":"Hahn","year":"2019","venue":"TACAS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hasanbeig et al., 2019] M. Hasanbeig, Yiannis Kantaros, A. Abate, D. Kroening, George Pappas, and I. Lee. Re- inforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees. In CDC, 2019.","title":"Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees","authors":"M Hasanbeig, A Yiannis Kantaros, D Abate, George Kroening, I Pappas, Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Henriques et al., 2012] David Henriques, João G. Martins, Paolo Zuliani, André Platzer, and Edmund M. Clarke. Sta- tistical model checking for markov decision processes. In QEST, 2012. Jiang et al., 2020] Yuqian Jiang, Sudarshanan Bharadwaj, Bo Wu, Rishi Shah, Ufuk Topcu, and Peter Stone. Temporal-logic-based reward shaping for continuing learning tasks. arXiv preprint: 2007.01498, 2020.","title":"Temporal-logic-based reward shaping for continuing learning tasks","authors":"David Henriques, G João, Paolo Martins, André Zuliani, Edmund M Platzer, Clarke, Jiang","year":"2012","venue":"QEST","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jothimurugan et al., 2019] Kishor Jothimurugan, R. Alur, and Osbert Bastani. A composable specification language for reinforcement learning tasks. In NeurlPS, 2019.","title":"A composable specification language for reinforcement learning tasks","authors":"R Kishor Jothimurugan, Osbert Alur, Bastani","year":"2019","venue":"NeurlPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kakade, 2003] Sham M. Kakade. On the Sample Complex- ity of Reinforcement Learning. PhD thesis, Gatsby Com- putational Neuroscience Unit, UCL, 2003.","title":"On the Sample Complexity of Reinforcement Learning","authors":"M Sham, Kakade","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kearns and Singh, 2002] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polyno- mial time. Machine Learning, 49(2), 2002.","title":"Near-optimal reinforcement learning in polynomial time","authors":"Michael Kearns, Satinder Singh","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kearns et al., 1999] Michael Kearns, Yishay Mansour, and Andrew Y. Ng. Approximate planning in large pomdps via reusable trajectories. In NIPS, 1999.","title":"Approximate planning in large pomdps via reusable trajectories","authors":"Michael Kearns, Yishay Mansour, Andrew Y Ng","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kober et al., 2013] Jens Kober, J. Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The Inter- national Journal of Robotics Research, 32, 2013.","title":"Reinforcement learning in robotics: A survey","authors":"Jens Kober, J Bagnell, Jan Peters","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2017] Xiao Li, C. Vasile, and C. Belta. Reinforce- ment learning with temporal logic rewards. IROS, 2017.","title":"Reinforcement learning with temporal logic rewards","authors":"Xiao Li, C Vasile, C Belta","year":"2017","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Littman et al., 2017] Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Isbell, Min Wen, and James MacGlashan. Environment-independent task specifications via gltl. arXiv preprint: 1704.04341, 2017.","title":"Environment-independent task specifications via gltl","authors":"L Michael, Ufuk Littman, Jie Topcu, Charles Fu, Min Isbell, James Wen, Macglashan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Manna and Pnueli, 1987] Zohar Manna and Amir Pnueli. A hierarchy of temporal properties. In PODC, 1987.","title":"A hierarchy of temporal properties","authors":"Pnueli Manna, Zohar Manna, Amir Pnueli","year":"1987","venue":"PODC","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pnueli, 1977] Amir Pnueli. The temporal logic of programs. In FOCS, 1977.","title":"The temporal logic of programs","authors":"Amir Pnueli, Pnueli","year":"1977","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes-Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.","title":"Markov Decision Processes-Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sadigh et al., 2014] Dorsa Sadigh, Eric S. Kim, Samuel Coogan, S. Shankar Sastry, and Sanjit A. Seshia. A learn- ing based approach to control synthesis of markov decision processes for linear temporal logic specifications. In CDC, 2014.","title":"A learning based approach to control synthesis of markov decision processes for linear temporal logic specifications","authors":"Dorsa Sadigh, Eric S Kim, S Shankar Samuel Coogan, Sanjit A Sastry, Seshia","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schwarting et al., 2018] Wilko Schwarting, Javier Alonso- Mora, and Daniela Rus. Planning and decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems, 1, 2018.","title":"Planning and decision-making for autonomous vehicles","authors":"Wilko Schwarting, Javier Alonso-Mora, Daniela Rus","year":"2018","venue":"Annual Review of Control, Robotics, and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Strehl et al., 2006] Alexander Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael Littman. Pac model-free reinforcement learning. In ICML, 2006.","title":"Pac model-free reinforcement learning","authors":"Alexander Strehl, Lihong Li, Eric Wiewiora, John Langford, Michael Littman","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Temizer et al., 2010] Selim Temizer, Mykel Kochender- fer, Leslie Kaelbling, Tomas Lozano-Perez, and James Kuchar. Collision avoidance for unmanned aircraft using markov decision processes. In AIAA GNC, 2010.","title":"Collision avoidance for unmanned aircraft using markov decision processes","authors":"Selim Temizer, Mykel Kochenderfer, Leslie Kaelbling, Tomas Lozano-Perez, James Kuchar","year":"2010","venue":"AIAA GNC","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Valiant, 1984] L. G. Valiant. A theory of the learnable. Communications of the ACM, 27(11), 1984.","title":"A theory of the learnable","authors":"L G Valiant","year":"1984","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}