{"type":"ScholarlyArticle","title":"Reinforcement Learning with Option Machines","authors":[{"name":"Floris den Hengst","slug":"floris-den-hengst","given":"Floris","family":"den Hengst","affiliation":"Vrije Universiteit Amsterdam"},{"name":"Vincent Francois-Lavet","slug":"vincent-francois-lavet","given":"Vincent","family":"François-Lavet","affiliation":"Vrije Universiteit Amsterdam"},{"name":"Mark Hoogendoorn","slug":"mark-hoogendoorn","given":"Mark","family":"Hoogendoorn","affiliation":"Vrije Universiteit Amsterdam"},{"name":"Frank van Harmelen","slug":"frank-van-harmelen","given":"Frank","family":"van Harmelen","affiliation":"Vrije Universiteit Amsterdam"}],"abstract":"Reinforcement learning (RL) is a powerful framework for learning complex behaviors, but lacks adoption in many settings due to sample size requirements. We introduce a framework for increasing sample efficiency of RL algorithms. Our approach focuses on optimizing environment rewards with high-level instructions. These are modeled as a high-level controller over temporally extended actions known as options. These options can be looped, interleaved and partially ordered with a rich language for high-level instructions. Crucially, the instructions may be underspecified in the sense that following them does not guarantee high reward in the environment. We present an algorithm for control with these so-called option machines (OMs), discuss option selection for the partially ordered case and describe an algorithm for learning with OMs. We compare our approach in zero-shot, single- and multi-task settings in an environment with fully specified and underspecified instructions. We find that OMs perform significantly better than or comparable to the state-of-art in all environments and learning settings.","doi":"10.24963/ijcai.2022/403","url":"https://ijcai.org/proceedings/2022/403","pdf":"https://ijcai.org/proceedings/2022/0403.pdf","markdown":"https://ijcai.org/proceedings/2022/403.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/403","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":403,"pages":{"first":2909,"last":2915},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Planning with Incomplete Information","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987091","length_sec":54,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985234","length_sec":629,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Andreas et al., 2017] J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. In ICML, pages 166-175. PMLR, 2017.","title":"Modular multitask reinforcement learning with policy sketches","authors":"J Andreas, D Klein, S Levine","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In ICML, volume 26, pages 41-48, 2009.","title":"Curriculum learning","authors":"Y Bengio, J Louradour, R Collobert, J Weston","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brafman et al., 2018] R. Brafman, G. De Giacomo, and F. Patrizi. Ltlf/ldlf non-markovian rewards. In AAAI, vol- ume 32. AAAI, 2018.","title":"Ltlf/ldlf non-markovian rewards","authors":"R Brafman, G De Giacomo, F Patrizi","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Camacho et al., 2019] A. Camacho, R. Icarte, T. Klassen, R. Valenzano, and S. McIlraith. Ltl and beyond: Formal languages for reward function specification in reinforce- ment learning. In IJCAI, pages 6065-6073, 2019.","title":"Ltl and beyond: Formal languages for reward function specification in reinforcement learning","authors":"A Camacho, R Icarte, T Klassen, R Valenzano, S Mcilraith","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/840"},{"position":5,"raw":"[den Hengst et al., 2020] F. den Hengst, E.M. Grua, A. El Hassouni, and M. Hoogendoorn. Reinforcement learning for personalization: A systematic literature review. Data Science, 3(2):107-147, 2020.","title":"Reinforcement learning for personalization: A systematic literature review","authors":"F Hengst, E M Grua, A El Hassouni, M Hoogendoorn","year":"2020","venue":"Data Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[den Hengst et al., 2022] F. den Hengst, V. Franc ¸ois-Lavet, M. Hoogendoorn, and F. van Harmelen. Planning for po- tential: efficient safe reinforcement learning. Machine Learning, pages 1-20, 2022.","title":"Planning for potential: efficient safe reinforcement learning","authors":"F Hengst, V Franc ¸ois-Lavet, M Hoogendoorn, F Van Harmelen","year":"2022","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dietterich, 2000] T.G. Dietterich. Hierarchical reinforce- ment learning with the maxq value function decomposi- tion. JAIR, 13:227-303, 2000.","title":"Hierarchical reinforcement learning with the maxq value function decomposition","authors":"T G Dietterich","year":"2000","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dulac-Arnold et al., 2019] G. Dulac-Arnold, D. Mankowitz, and T. Hester. Challenges of real-world reinforcement learning. arXiv:1904.12901, 2019.","title":"Challenges of real-world reinforcement learning","authors":"G Dulac-Arnold, D Mankowitz, T Hester","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fu and Topcu, 2014] J. Fu and U. Topcu. Probably approx- imately correct mdp learning and control with temporal logic constraints. In Robotics: Science and Systems, num- ber 10, 2014.","title":"Probably approximately correct mdp learning and control with temporal logic constraints","authors":"J Fu, U Topcu","year":"2014","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grounds and Kudenko, 2005] M. Grounds and D. Kudenko. Combining reinforcement learning with symbolic plan- ning. In AAMAS, pages 75-86. Springer, 2005.","title":"Combining reinforcement learning with symbolic planning","authors":"M Grounds, D Kudenko","year":"2005","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grzes and Kudenko, 2008] M. Grzes and D. Kudenko. Plan-based reward shaping for reinforcement learning. In IntelliSys, volume 2, pages 10-22. IEEE, 2008.","title":"Plan-based reward shaping for reinforcement learning","authors":"M Grzes, D Kudenko","year":"2008","venue":"IntelliSys","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Illanes et al., 2020] L. Illanes, X. Yan, R. Icarte, and S. McIlraith. Symbolic plans as high-level instructions for reinforcement learning. In ICAPS, volume 30, pages 540- 550, 2020.","title":"Symbolic plans as high-level instructions for reinforcement learning","authors":"L Illanes, X Yan, R Icarte, S Mcilraith","year":"2020","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Konda and Tsitsiklis, 2000] V. Konda and J. Tsitsiklis. Actor-critic algorithms. In NIPS, pages 1008-1014, 2000.","title":"Actor-critic algorithms","authors":"V Konda, J Tsitsiklis","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Leonetti et al., 2012] M. Leonetti, L. Iocchi, and F. Pa- trizi. Automatic generation and learning of finite-state controllers. In AIMSA, pages 135-144. Springer, 2012.","title":"Automatic generation and learning of finite-state controllers","authors":"M Leonetti, L Iocchi, F Patrizi","year":"2012","venue":"AIMSA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Leonetti et al., 2016] M. Leonetti, L. Iocchi, and P. Stone. A synthesis of automated planning and reinforcement learn- ing for efficient, robust decision-making. Artificial Intelli- gence, 241:103-130, 2016.","title":"A synthesis of automated planning and reinforcement learning for efficient, robust decision-making","authors":"M Leonetti, L Iocchi, P Stone","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] X. Li, C. Vasile, and C. Belta. Reinforce- ment learning with temporal logic rewards. In IROS, pages 3834-3839. IEEE/RSJ, 2017.","title":"Reinforcement learning with temporal logic rewards","authors":"Li","year":"2017","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lyu et al., 2019] D. Lyu, F. Yang, B. Liu, and S. Gustafson. Sdrl: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In AAAI, vol- ume 33, pages 2970-2977, 2019.","title":"Sdrl: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning","authors":"D Lyu, F Yang, B Liu, S Gustafson","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mealy, 1955] G.H. Mealy. A method for synthesizing se- quential circuits. The Bell System Technical Journal, 34(5):1045-1079, 1955.","title":"A method for synthesizing sequential circuits","authors":"G H Mealy","year":"1955","venue":"The Bell System Technical Journal","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Michaud and Colange, 2018] T. Michaud and M. Colange. Reactive synthesis from ltl specification with spot. In 7th Workshop on Synthesis, SYNT@ CAV, 2018.","title":"Reactive synthesis from ltl specification with spot","authors":"T Michaud, M Colange","year":"2018","venue":"7th Workshop on Synthesis","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ng et al., 1999] A.Y. Ng, D. Harada, and S. Russell. Pol- icy invariance under reward transformations: Theory and application to reward shaping. In ICML, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"A Y Ng, D Harada, S Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Parr and Russell, 1998] R. Parr and S. Russell. Reinforce- ment learning with rhierarchies of machines. NIPS, pages 1043-1049, 1998.","title":"Reinforcement learning with rhierarchies of machines","authors":"R Parr, S Russell","year":"1998","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ryan, 2002] M. Ryan. Using abstract models of behaviours to automatically generate reinforcement learning hierar- chies. In ICML, volume 2, pages 522-529, 2002.","title":"Using abstract models of behaviours to automatically generate reinforcement learning hierarchies","authors":"M Ryan","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sadigh et al., 2014] D. Sadigh, E.S. Kim, S. Coogan, S.S. Sastry, and S. Seshia. A learning based approach to control synthesis of markov decision processes for linear tempo- ral logic specifications. In CDC, pages 1091-1096. IEEE, 2014.","title":"A learning based approach to control synthesis of markov decision processes for linear temporal logic specifications","authors":"D Sadigh, E S Kim, S Coogan, S S Sastry, S Seshia","year":"2014","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Singh, 1992] S. Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8(3):323-339, 1992.","title":"Transfer of learning by composing solutions of elemental sequential tasks","authors":"S Singh","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 2018] R.S. Sutton and A. Barto. Rein- forcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"R S Sutton, A Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 1999] R.S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for tempo- ral abstraction in reinforcement learning. Artificial Intelli- gence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"R S Sutton, D Precup, S Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Toro Icarte et al., 2018a] R. Toro Icarte, T. Klassen, R. Valenzano, and S. McIlraith. Teaching multiple tasks to an rl agent using ltl. In AAMAS, pages 452-461, 2018.","title":"Teaching multiple tasks to an rl agent using ltl","authors":"Toro Icarte","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Toro Icarte et al., 2018b] R. Toro Icarte, T. Klassen, R. Valenzano, and S. McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In ICML, volume 35, pages 2107-2116. PMLR, 2018.","title":"Using reward machines for high-level task specification and decomposition in reinforcement learning","authors":"Toro Icarte","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Williams, 1992] R.J. Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-256, 1992.","title":"Simple statistical gradientfollowing algorithms for connectionist reinforcement learning","authors":"R J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2018] F. Yang, D. Lyu, B. Liu, and S. Gustafson. Peorl: integrating symbolic planning and hierarchical reinforcement learning for robust decision-making. In IJCAI, pages 4860-4866, 2018.","title":"Peorl: integrating symbolic planning and hierarchical reinforcement learning for robust decision-making","authors":"F Yang, D Lyu, B Liu, S Gustafson","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/675"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}