{"type":"ScholarlyArticle","title":"LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning","authors":[{"name":"Alberto Camacho","slug":"alberto-camacho","given":"Alberto","family":"Camacho","affiliation":"Department of Computer Science , University of Toronto , Toronto , Canada; Vector Institute , Toronto , Canada"},{"name":"Rodrigo Toro Icarte","slug":"rodrigo-toro-icarte","given":"Rodrigo Toro","family":"Icarte","affiliation":"Department of Computer Science , University of Toronto , Toronto , Canada; Vector Institute , Toronto , Canada"},{"name":"Toryn Q. Klassen","slug":"toryn-q-klassen","given":"Toryn Q.","family":"Klassen","affiliation":"Department of Computer Science , University of Toronto , Toronto , Canada"},{"name":"Richard Valenzano","slug":"richard-valenzano","given":"Richard","family":"Valenzano","affiliation":"Element AI , Toronto , Canada"},{"name":"Sheila A. McIlraith","slug":"sheila-a-mcilraith","given":"Sheila A.","family":"McIlraith","affiliation":"Department of Computer Science , University of Toronto , Toronto , Canada; Vector Institute , Toronto , Canada"}],"abstract":"In Reinforcement Learning (RL), an agent is guided by the rewards it receives from the reward function. Unfortunately, it may take many interactions with the environment to learn from sparse rewards, and it can be challenging to specify reward functions that reflect complex reward-worthy behavior. We propose using reward machines (RMs), which are automata-based representations that expose reward function structure, as a normal form representation for reward functions. We show how specifications of reward in various formal languages, including LTL and other regular languages, can be automatically translated into RMs, easing the burden of complex reward function specification. We then show how the exposed structure of the reward function can be exploited by tailored q-learning algorithms and automated reward shaping techniques in order to improve the sample efficiency of reinforcement learning methods. Experiments show that these RM-tailored techniques significantly outperform state-of-the-art (deep) RL algorithms, solving problems that otherwise cannot reasonably be solved by existing approaches.","doi":"10.24963/ijcai.2019/840","url":"https://ijcai.org/proceedings/2019/840","pdf":"https://ijcai.org/proceedings/2019/0840.pdf","markdown":"https://ijcai.org/proceedings/2019/840.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/840","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Understanding Intelligence and Human-level AI in the New Machine Learning era","track":"Understanding Intelligence and Human-level AI in the New Machine Learning era","number":840,"pages":{"first":6065,"last":6073},"keywords":["Special Track on Understanding Intelligence and Human-level AI in the New Machine Learning era: Knowledge representations for Learning (Special Track on Human AI and Machine Learning)"],"videos":[],"references":[{"position":1,"raw":"[Andreas et al., 2017] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with pol- icy sketches. In ICML, pages 166-175, 2017.","title":"Modular multitask reinforcement learning with policy sketches","authors":"Jacob Andreas, Dan Klein, Sergey Levine","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacchus et al., 1996] Fahiem Bacchus, Craig Boutilier, and Adam J. Grove. Rewarding behaviors. In AAAI, pages 1160- 1167, 1996.","title":"Rewarding behaviors","authors":"Bacchus","year":"1996","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bacchus et al., 1997] Fahiem Bacchus, Craig Boutilier, and Adam J. Grove. Structured solution methods for non- Markovian decision processes. In AAAI, pages 112-117, 1997.","title":"Structured solution methods for non-Markovian decision processes","authors":"Bacchus","year":"1997","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baier and McIlraith, 2006] Jorge A. Baier and Sheila A. McIl- raith. Planning with temporally extended goals using heuristic search. In ICAPS, pages 342-345, 2006.","title":"Planning with temporally extended goals using heuristic search","authors":"Jorge A Baier, Sheila A Mcilraith","year":"2006","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baier et al., 2007] Jorge A. Baier, Christian Fritz, and Sheila A. McIlraith. Exploiting procedural domain control knowledge in state-of-the-art planners. In ICAPS, pages 26- 33, 2007.","title":"Exploiting procedural domain control knowledge in state-of-the-art planners","authors":"Jorge A Baier, Christian Fritz, Sheila A Mcilraith","year":"2007","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Baier et al., 2008] Jorge A. Baier, Christian Fritz, Meghyn Bi- envenu, and Sheila McIlraith. Beyond classical planning: Procedural control knowledge and preferences in state-of-the- art planners. In AAAI, Nectar Track, pages 1509-1512, 2008.","title":"Beyond classical planning: Procedural control knowledge and preferences in state-of-theart planners","authors":"Jorge A Baier, Christian Fritz, Meghyn Bienvenu, Sheila Mcilraith","year":"2008","venue":"AAAI, Nectar Track","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brafman et al., 2018] Ronen I. Brafman, Giuseppe De Gia- como, and Fabio Patrizi. LTLf/LDLf non-Markovian rewards. In AAAI, pages 1771-1778, 2018.","title":"LTLf/LDLf non-Markovian rewards","authors":"Ronen I Brafman, Giuseppe De, Giacomo, Fabio Patrizi","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Camacho et al., 2017] Alberto Camacho, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. Non-Markovian rewards ex- pressed in LTL: guiding search via reward shaping. In SOCS, pages 159-160, 2017.","title":"Non-Markovian rewards expressed in LTL: guiding search via reward shaping","authors":"Alberto Camacho, Oscar Chen, Scott Sanner, Sheila A Mcilraith","year":"2017","venue":"SOCS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Camacho et al., 2018] Alberto Camacho, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. Non-markovian rewards expressed in LTL: Guiding search via reward shaping (ex- tended version). In GoalsRL, a workshop collocated with ICML/IJCAI/AAMAS, 2018.","title":"Non-markovian rewards expressed in LTL: Guiding search via reward shaping (extended version)","authors":"Alberto Camacho, Oscar Chen, Scott Sanner, Sheila A Mcilraith","year":"2018","venue":"GoalsRL, a workshop collocated with ICML/IJCAI/AAMAS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[De Giacomo and Vardi, 2013] Giuseppe De Giacomo and Moshe Y. Vardi. Linear temporal logic and linear dynamic logic on finite traces. In IJCAI, pages 854-860, 2013.","title":"Linear temporal logic and linear dynamic logic on finite traces","authors":"De Giacomo, Giuseppe De Vardi, Moshe Y Giacomo, Vardi","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Duret-Lutz et al., 2016] Alexandre Duret-Lutz, Alexandre Lewkowicz, Amaury Fauchille, Thibaud Michaud, Etienne Renault, and Laurent Xu. Spot 2.0 -a framework for LTL and ω-automata manipulation. In ATVA, volume 9938 of LNCS, pages 122-129. Springer, October 2016.","title":"Spot 2.0 -a framework for LTL and ω-automata manipulation","authors":"Alexandre Duret-Lutz, Alexandre Lewkowicz, Amaury Fauchille, Thibaud Michaud, Etienne Renault, Laurent Xu","year":"2016","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Emerson, 1990] E. Allen Emerson. Temporal and modal logic. In Handbook of Theoretical Computer Science, volume B, pages 995-1072. Elsevier, 1990.","title":"Temporal and modal logic","authors":"E, Allen Emerson","year":"1990","venue":"Handbook of Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hopcroft and Ullman, 1979] John E. Hopcroft and Jeffrey D. Ullman. Introduction to Automata Theory, Languages and Computation. Addison-Wesley, 1979.","title":"Introduction to Automata Theory, Languages and Computation","authors":"John E Hopcroft, Jeffrey D Ullman","year":"1979","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karpathy, 2015] Andrej Karpathy. REINFORCEjs: Water- World demo. Retrieved from http://cs.stanford.edu/people/ karpathy/reinforcejs/waterworld.html, 2015.","title":"REINFORCEjs: Water-World demo","authors":"Karpathy","year":"2015","venue":null,"doi":null,"url":"http://cs.stanford.edu/people/karpathy/reinforcejs/waterworld.html,2015","internal_url":null},{"position":15,"raw":"[Kupferman and Vardi, 2001] Orna Kupferman and Moshe Y. Vardi. Model checking of safety properties. Formal Meth- ods in System Design, 19(3):291-314, 2001.","title":"Model checking of safety properties","authors":"Vardi ; ] Orna Kupferman, Moshe Y Kupferman, Vardi","year":"2001","venue":"Formal Methods in System Design","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lamport, 2002] Leslie Lamport. Specifying Systems, The TLA+ Language and Tools for Hardware and Software En- gineers. Addison-Wesley, 2002.","title":"Specifying Systems, The TLA+ Language and Tools for Hardware and Software Engineers","authors":"Leslie Lamport","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2018] Tyler Lu, Dale Schuurmans, and Craig Boutilier. Non-delusional Q-learning and value-iteration. In NeurIPS, pages 9971-9981, 2018.","title":"Non-delusional Q-learning and value-iteration","authors":"Lu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mealy, 1955] George H Mealy. A method for synthesiz- ing sequential circuits. The Bell System Technical Journal, 34(5):1045-1079, Sep. 1955.","title":"A method for synthesizing sequential circuits","authors":"H George, Mealy","year":"1955","venue":"The Bell System Technical Journal","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-level control through deep re- inforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ng et al., 1999] Andrew Y. Ng, Daishi Harada, and Stuart Rus- sell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 3, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Y Ng, Daishi Harada, Stuart Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sohrabi et al., 2011] Shirin Sohrabi, Jorge A. Baier, and Sheila A. McIlraith. Preferred explanations: Theory and gen- eration via planning. In AAAI-11, pages 261-267, 2011.","title":"Preferred explanations: Theory and generation via planning","authors":"Shirin Sohrabi, Jorge A Baier, Sheila A Mcilraith","year":"2011","venue":"AAAI-11","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, second edition, 2018.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thiébaux et al., 2006] Sylvie Thiébaux, Charles Gretton, John K. Slaney, David Price, and Froduald Kabanza. Decision-theoretic planning with non-Markovian rewards. Journal of Artificial Intelligence Research, 25:17-74, 2006.","title":"Decision-theoretic planning with non-Markovian rewards","authors":"Sylvie Thiébaux, Charles Gretton, John K Slaney, David Price, Froduald Kabanza","year":"2006","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thompson, 1968] Ken Thompson. Regular expression search algorithm. Communications of the ACM, 11(6):419-422, 1968.","title":"Regular expression search algorithm","authors":"Ken Thompson, Thompson","year":"1968","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Toro Icarte et al., 2018a] Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Anthony Valenzano, and Sheila A. McIl- raith. Teaching multiple tasks to an RL agent using LTL. In AAMAS, pages 452-461, 2018.","title":"McIlraith. Teaching multiple tasks to an RL agent using LTL","authors":"Toro Icarte","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Toro Icarte et al., 2018b] Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Anthony Valenzano, and Sheila A. McIl- raith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In ICML, pages 2112-2121, 2018.","title":"McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning","authors":"Toro Icarte","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Triantafillou et al., 2015] Eleni Triantafillou, Jorge A. Baier, and Sheila A. McIlraith. A unifying framework for planning with LTL and regular expressions. In MOCHAP, a workshop collocated with ICAPS, pages 23-31, 2015.","title":"A unifying framework for planning with LTL and regular expressions","authors":"Eleni Triantafillou, Jorge A Baier, Sheila A Mcilraith","year":"2015","venue":"MOCHAP, a workshop collocated with ICAPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q- learning. In AAAI, pages 2094-2100, 2016.","title":"Deep reinforcement learning with double Qlearning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Watkins and Dayan, 1992] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.","title":"Q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2017] Shufang Zhu, Lucas M. Tabajara, Jianwen Li, Geguang Pu, and Moshe Y. Vardi. Symbolic LTLf synthesis. In IJCAI, pages 1362-1369, 2017.","title":"Symbolic LTLf synthesis","authors":"Shufang Zhu, Lucas M Tabajara, Jianwen Li, Geguang Pu, Moshe Y Vardi","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}