{"type":"ScholarlyArticle","title":"Successor Options: An Option Discovery Framework for Reinforcement Learning","authors":[{"name":"Rahul Ramesh","slug":"rahul-ramesh","given":"Rahul","family":"Ramesh","affiliation":""},{"name":"Manan Tomar","slug":"manan-tomar","given":"Manan","family":"Tomar","affiliation":"Department of Engineering Design , Indian Institute of Technology Madras"},{"name":"Balaraman Ravindran","slug":"balaraman-ravindran","given":"Balaraman","family":"Ravindran","affiliation":"Department of Computer Science & Engineering , Indian Institute of Technology Madras"}],"abstract":"The options framework in reinforcement learning models the notion of a skill or a temporally extended sequence of actions. The discovery of a reusable set of skills has typically entailed building options, that navigate to bottleneck states. In this work, we instead adopt a complementary approach, where we attempt to discover options that navigate to landmark states. These states are prototypical representatives of well-connected regions and can hence access the associated region with relative ease. In this work, we propose Successor Options, which leverages Successor representations to build a model of the state space. The intra-option policies are learnt using a novel pseudo-reward and the model scales to high-dimensional spaces since it does not construct an explicit graph of the entire state space. Additionally, we also propose an Incremental Successor Options model that iterates between constructing Successor representations and building options, which is useful when robust Successor representations cannot be built solely from primitive actions. We demonstrate the efficacy of our approach on a collection of grid-worlds, and on the high-dimensional robotic control environment of Fetch.","doi":"10.24963/ijcai.2019/458","url":"https://ijcai.org/proceedings/2019/458","pdf":"https://ijcai.org/proceedings/2019/0458.pdf","markdown":"https://ijcai.org/proceedings/2019/458.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/458","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":458,"pages":{"first":3304,"last":3310},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Arthur and Vassilvitskii, 2007] David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seed- ing. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035. So- ciety for Industrial and Applied Mathematics, 2007.","title":"David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding","authors":"Vassilvitskii Arthur","year":"2007","venue":"Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barreto et al., 2017] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Has- selt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Informa- tion Processing Systems, pages 4055-4065, 2017.","title":"Successor features for transfer in reinforcement learning","authors":"André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, David Hado P Van Hasselt, Silver","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dayan, 1993] Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-624, 1993.","title":"Improving generalization for temporal difference learning: The successor representation","authors":"Peter Dayan","year":"1993","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kulkarni et al., 2016a] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Ad- vances in neural information processing systems, pages 3675-3683, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"Kulkarni","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kulkarni et al., 2016b] Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396, 2016.","title":"Deep successor reinforcement learning","authors":"Kulkarni","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lakshminarayanan et al., 2016] Aravind S Lakshmi- narayanan, Ramnandan Krishnamurthy, Peeyush Kumar, and Balaraman Ravindran. Option discovery in hier- archical reinforcement learning using spatio-temporal clustering. arXiv preprint arXiv:1605.05359, 2016.","title":"Option discovery in hierarchical reinforcement learning using spatio-temporal clustering","authors":"Ramnandan Aravind S Lakshminarayanan, Peeyush Krishnamurthy, Balaraman Kumar, Ravindran","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Machado et al., 2017a] Marlos C Machado, Marc G Belle- mare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. arXiv preprint arXiv:1703.00956, 2017.","title":"A laplacian framework for option discovery in reinforcement learning","authors":"Machado","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Machado et al., 2017b] Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv:1710.11089, 2017.","title":"Eigenoption discovery through the deep successor representation","authors":"Machado","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Machado et al., 2018] Marlos C Machado, Marc G Belle- mare, and Michael Bowling. Count-based explo- ration with the successor representation. arXiv preprint arXiv:1807.11622, 2018.","title":"Count-based exploration with the successor representation","authors":"C Marlos, Marc G Machado, Michael Bellemare, Bowling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[McGovern and Barto, 2001] Amy McGovern and An- drew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. 2001.","title":"Automatic discovery of subgoals in reinforcement learning using diverse density","authors":"Barto Mcgovern, Amy Mcgovern, Andrew G Barto","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Menache et al., 2002] Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut-dynamic discovery of sub-goals in reinforcement learning. In European Conference on Machine Learning, pages 295-306. Springer, 2002.","title":"Q-cut-dynamic discovery of sub-goals in reinforcement learning","authors":"Ishai Menache, Shie Mannor, Nahum Shimkin","year":"2002","venue":"European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, and others. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Others","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Moore et al., 1999] Andrew W Moore, L Baird, and LP Kaelbling. Multi-value-functions: E cient automatic action hierarchies for multiple goal mdps. 1999.","title":"Multi-value-functions: E cient automatic action hierarchies for multiple goal mdps","authors":"L Andrew W Moore, Baird, Kaelbling","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Plappert et al., 2018] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.","title":"Multi-goal reinforcement learning: Challenging robotics environments and request for research","authors":"Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob Mcgrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, Wojciech Zaremba","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Puterman, 1994] Martin L Puterman. Markov decision pro- cesses: Discrete stochastic dynamic programming. 1994.","title":"Markov decision processes: Discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[S ¸ims ¸ek and Barto, 2004] Özgür S ¸ims ¸ek and Andrew G Barto. Using relative novelty to identify useful tempo- ral abstractions in reinforcement learning. In Proceed- ings of the twenty-first international conference on Ma- chine learning, page 95. ACM, 2004.","title":"Using relative novelty to identify useful temporal abstractions in reinforcement learning","authors":"; ] Barto, S Özgür, Andrew G Barto","year":"2004","venue":"Proceedings of the twenty-first international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[S ¸ims ¸ek and Barto, 2009] Özgür S ¸ims ¸ek and Andrew G Barto. Skill characterization based on betweenness. In Advances in neural information processing systems, pages 1497-1504, 2009.","title":"Skill characterization based on betweenness","authors":"; ] Barto, S Özgür, Andrew G Barto","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Özgür S ¸ims ¸ek, Alicia P. Wolfe, and Andrew G. Barto. Identifying useful subgoals in reinforce- ment learning by local graph partitioning. pages 816-823. ACM Press, 2005.","title":"Identifying useful subgoals in reinforcement learning by local graph partitioning","authors":"Alicia P Özgür S ¸ims ¸ek, Andrew G Wolfe, Barto","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. FeU- dal Networks for Hierarchical Reinforcement Learning. arXiv:1703.01161 [cs], March 2017.","title":"FeUdal Networks for Hierarchical Reinforcement Learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Nicolas Schaul, Max Heess, David Jaderberg, Koray Silver, Kavukcuoglu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}