{"type":"ScholarlyArticle","title":"The Expected-Length Model of Options","authors":[{"name":"David Abel","slug":"david-abel","given":"David","family":"Abel","affiliation":"Brown University"},{"name":"John Winder","slug":"john-winder","given":"John","family":"Winder","affiliation":"University of Maryland , Baltimore County"},{"name":"Marie desJardins","slug":"marie-desjardins","given":"Marie","family":"desJardins","affiliation":"Simmons University"},{"name":"Michael Littman","slug":"michael-littman","given":"Michael","family":"Littman","affiliation":"Brown University"}],"abstract":"Effective options can make reinforcement learning easier by enhancing an agent's ability to both explore in a targeted manner and plan further into the future. However, learning an appropriate model of an option's dynamics in hard, requiring estimating a highly parameterized probability distribution. This paper introduces and motivates the Expected-Length Model (ELM) for options, an alternate model for transition dynamics. We prove ELM is a (biased) estimator of the traditional Multi-Time Model (MTM), but provide a non-vacuous bound on their deviation. We further prove that, in stochastic shortest path problems, ELM induces a value function that is sufficiently similar to the one induced by MTM, and is thus capable of supporting near-optimal behavior. We explore the practical utility of this option model experimentally, finding consistent support for the thesis that ELM is a suitable replacement for MTM. In some cases, we find ELM leads to more sample efficient learning, especially when options are arranged in a hierarchy.","doi":"10.24963/ijcai.2019/270","url":"https://ijcai.org/proceedings/2019/270","pdf":"https://ijcai.org/proceedings/2019/0270.pdf","markdown":"https://ijcai.org/proceedings/2019/270.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/270","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":270,"pages":{"first":1951,"last":1958},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Model-Based Reasoning","Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Hierarchical planning"],"videos":[],"references":[{"position":1,"raw":"[Asadi et al., 2018] Kavosh Asadi, Dipendra Misra, and Michael L Littman. Lipschitz continuity in model-based reinforcement learning. ICML, 2018.","title":"Lipschitz continuity in model-based reinforcement learning","authors":"Asadi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pages 1726-1734, 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brafman and Tennenholtz, 2002] Ronen I Brafman and Moshe Tennenholtz. R-MAX: A general polynomial time algorithm for near-optimal reinforcement learning. JMLR, 3(Oct):213-231, 2002.","title":"R-MAX: A general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brunskill and Li, 2014] Emma Brunskill and Lihong Li. PAC-inspired option discovery in lifelong reinforcement learning. In ICML, pages 316-324, 2014.","title":"PAC-inspired option discovery in lifelong reinforcement learning","authors":"Li ; Brunskill, Emma Brunskill, Lihong Li","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dietterich, 2000] Thomas G Dietterich. Hierarchical rein- forcement learning with the MAXQ value function decom- position. JAIR, 13:227-303, 2000.","title":"Hierarchical reinforcement learning with the MAXQ value function decomposition","authors":"G Thomas, Dietterich","year":"2000","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fruit and Lazaric, 2017] Ronan Fruit and Alessandro Lazaric. Exploration-exploitation in MDPs with options. In AISTATS, pages 576-584, 2017.","title":"Ronan Fruit and Alessandro Lazaric. Exploration-exploitation in MDPs with options","authors":"Lazaric Fruit","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gopalan et al., 2017] Nakul Gopalan, Marie desJardins, Michael L Littman, James MacGlashan, Shawn Squire, Stefanie Tellex, John Winder, and Lawson LS Wong. Plan- ning with abstract Markov decision processes. In ICAPS, 2017.","title":"Planning with abstract Markov decision processes","authors":"Nakul Gopalan, Marie Desjardins, James Michael L Littman, Shawn Macglashan, Stefanie Squire, John Tellex, Lawson Ls Winder, Wong","year":"2017","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Harutyunyan et al., 2018] Anna Harutyunyan, Peter Vrancx, Pierre-Luc Bacon, Doina Precup, and Ann Nowé. Learning with options that terminate off-policy. In AAAI, 2018.","title":"Learning with options that terminate off-policy","authors":"Anna Harutyunyan, Peter Vrancx, Pierre-Luc Bacon, Doina Precup, Ann Nowé","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jong and Stone, 2008] Nicholas K Jong and Peter Stone. Hierarchical model-based reinforcement learning: R- MAX+MAXQ. In ICML, pages 432-439. ACM, 2008.","title":"Hierarchical model-based reinforcement learning: R-MAX+MAXQ","authors":"K Nicholas, Peter Jong, Stone","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jong, 2010] Nicholas K Jong. Structured Exploration for Reinforcement Learning. PhD thesis, U. of Texas, 2010.","title":"Structured Exploration for Reinforcement Learning","authors":"Jong Nicholas","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kearns and Singh, 2002] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polyno- mial time. Machine Learning, 2002.","title":"Near-optimal reinforcement learning in polynomial time","authors":"Michael Kearns, Satinder Singh","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Konidaris and Barto, 2007] George Konidaris and An- drew G Barto. Building portable options: Skill transfer in reinforcement learning. In IJCAI, 2007.","title":"George Konidaris and Andrew G Barto. Building portable options: Skill transfer in reinforcement learning","authors":"Barto Konidaris","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Konidaris and Barto, 2009] George Konidaris and An- drew G Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In NeurIPS, pages 1015-1023, 2009.","title":"George Konidaris and Andrew G Barto. Skill discovery in continuous reinforcement learning domains using skill chaining","authors":"Barto Konidaris","year":"2009","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Konidaris et al., 2018] George Konidaris, Leslie Pack Kael- bling, and Tomas Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. JAIR, 61:215-289, 2018.","title":"George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning","authors":"Konidaris","year":"2018","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Littman and Szepesvári, 1996] Michael L Littman and Csaba Szepesvári. A generalized reinforcement-learning model: Convergence and applications. In ICML, volume 96, pages 310-318, 1996.","title":"A generalized reinforcement-learning model: Convergence and applications","authors":"L Michael, Csaba Littman, Szepesvári","year":"1996","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Littman et al., 1995] Michael L Littman, Thomas L Dean, and Leslie Pack Kaelbling. On the complexity of solving Markov decision problems. In UAI, pages 394-402, 1995.","title":"On the complexity of solving Markov decision problems","authors":"Thomas L Michael L Littman, Leslie Dean, Kaelbling Pack","year":"1995","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Machado et al., 2017] Marlos C Machado, Marc G Belle- mare, and Michael Bowling. A Laplacian framework for option discovery in reinforcement learning. ICML, 2017.","title":"A Laplacian framework for option discovery in reinforcement learning","authors":"C Marlos, Marc G Machado, Michael Bellemare, Bowling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mankowitz et al., 2014] Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Time-regularized interrupting options. In ICML, 2014.","title":"Time-regularized interrupting options","authors":"Timothy A Daniel J Mankowitz, Shie Mann, Mannor","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mankowitz et al., 2016] Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Adaptive skills adaptive parti- tions (ASAP). In NeurIPS, pages 1588-1596, 2016.","title":"Adaptive skills adaptive partitions (ASAP)","authors":"Timothy A Daniel J Mankowitz, Shie Mann, Mannor","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mann and Mannor, 2013] Timothy A Mann and Shie Man- nor. The advantage of planning with options. RLDM 2013, page 9, 2013.","title":"Timothy A Mann and Shie Mannor. The advantage of planning with options","authors":"Mannor Mann","year":"2013","venue":"RLDM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mann and Mannor, 2014] Timothy Mann and Shie Mannor. Scaling up approximate value iteration with options: Bet- ter policies with fewer iterations. In ICML, 2014.","title":"Scaling up approximate value iteration with options: Better policies with fewer iterations","authors":"Mannor Mann, Timothy Mann, Shie Mannor","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Parr, 1998] Ronald Edward Parr. Hierarchical Control and Learning for Markov Decision Processes. PhD thesis, Uni- versity of California, Berkeley, 1998.","title":"Hierarchical Control and Learning for Markov Decision Processes","authors":"Ronald Edward, Parr","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Precup and Sutton, 1997] Doina Precup and Richard S Sut- ton. Multi-time models for reinforcement learning. In ICML Workshop on Modelling in Reinforcement Learning, 1997.","title":"Multi-time models for reinforcement learning","authors":"Doina Precup, Richard S Sutton","year":"1997","venue":"ICML Workshop on Modelling in Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Precup and Sutton, 1998] Doina Precup and Richard S Sut- ton. Multi-time models for temporally abstract planning. In NeurIPS, pages 1050-1056, 1998.","title":"Multi-time models for temporally abstract planning","authors":"Doina Precup, Richard S Sutton","year":"1998","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Puterman, 2014] Martin L Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver and Ciosek, 2012] David Silver and Kamil Ciosek. Compositional planning using optimal option models. In ICML, volume 2, pages 1063-1070, 2012.","title":"Compositional planning using optimal option models","authors":"David Silver, Kamil Ciosek","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[S ¸ims ¸ek and Barto, 2004] Özgür S ¸ims ¸ek and Andrew G Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In ICML, page 95. ACM, 2004.","title":"Using relative novelty to identify useful temporal abstractions in reinforcement learning","authors":"; ] Barto, S Özgür, Andrew G Barto","year":"2004","venue":"ICML, page 95","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Singh et al., 2005] Satinder P Singh, Andrew G Barto, and Nuttapong Chentanez. Intrinsically motivated reinforce- ment learning. In NeurIPS, pages 1281-1288, 2005.","title":"Intrinsically motivated reinforcement learning","authors":"P Satinder, Andrew G Singh, Nuttapong Barto, Chentanez","year":"2005","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112:181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sutton, 1988] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9-44, 1988.","title":"Learning to predict by the methods of temporal differences","authors":"Richard S Sutton","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Talvitie, 2017] Erik Talvitie. Self-correcting models for model-based reinforcement learning. In AAAI, pages 2597-2603, 2017.","title":"Self-correcting models for model-based reinforcement learning","authors":"Erik Talvitie, Talvitie","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Topin et al., 2015] Nicholay Topin, Nicholas Haltmeyer, Shawn Squire, John Winder, James MacGlashan, and Marie desJardins. Portable option discovery for automated learning transfer in object-oriented Markov decision pro- cesses. In IJCAI, 2015.","title":"Portable option discovery for automated learning transfer in object-oriented Markov decision processes","authors":"Topin","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tseng, 1990] Paul Tseng. Solving H-horizon, stationary Markov decision problems in time proportional to log(H). Operations Research Letters, 9(5):287-297, 1990.","title":"Solving H-horizon, stationary Markov decision problems in time proportional to log(H)","authors":"Paul Tseng","year":"1990","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[White, 2017] Martha White. Unifying task specification in reinforcement learning. In ICML, pages 3742-3750, 2017.","title":"Unifying task specification in reinforcement learning","authors":"Martha White","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}