{"type":"ScholarlyArticle","title":"Option Transfer and SMDP Abstraction with Successor Features","authors":[{"name":"Dongge Han","slug":"dongge-han","given":"Dongge","family":"Han","affiliation":"University of Oxford , Department of Computer Science , Oxford , United Kingdom"},{"name":"Sebastian Tschiatschek","slug":"sebastian-tschiatschek","given":"Sebastian","family":"Tschiatschek","affiliation":"University of Vienna , Faculty of Computer Science , Vienna , Austria"}],"abstract":"Abstraction plays an important role in the generalisation of knowledge and skills and is key to sample efficient learning. In this work, we study joint temporal and state abstraction in reinforcement learning, where temporally-extended actions in the form of options induce temporal abstractions, while aggregation of similar states with respect to abstract options induces state abstractions. Many existing abstraction schemes ignore the interplay of state and temporal abstraction. Consequently, the considered option policies often cannot be directly transferred to new environments due to changes in the state space and transition dynamics. To address this issue, we propose a novel abstraction scheme building on successor features. This includes an algorithm for transferring abstract options across different environments and a state abstraction mechanism that allows us to perform efficient planning with the transferred options.","doi":"10.24963/ijcai.2022/421","url":"https://ijcai.org/proceedings/2022/421","pdf":"https://ijcai.org/proceedings/2022/0421.pdf","markdown":"https://ijcai.org/proceedings/2022/421.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/421","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":421,"pages":{"first":3036,"last":3042},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Hierarchical Planning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986999","length_sec":78,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985142","length_sec":622,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In International Conference on Machine Learning, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Ng Abbeel, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abel et al., 2016] David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning, pages 2915-2923. PMLR, 2016.","title":"Near optimal behavior via approximate state abstraction","authors":"David Abel, David Hershkowitz, Michael Littman","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Abel et al., 2020] David Abel, Nate Umbanhowar, Khimya Khetarpal, Dilip Arumugam, Doina Precup, and Michael Littman. Value preserving state-action abstractions. In International Conference on Artificial Intelligence and Statistics, pages 1639-1650. PMLR, 2020.","title":"Value preserving state-action abstractions","authors":"David Abel, Nate Umbanhowar, Khimya Khetarpal, Dilip Arumugam, Doina Precup, Michael Littman","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barreto et al., 2017] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in rein- forcement learning. In Advances in Neural Information Processing Systems, pages 4058-4068, 2017.","title":"Successor features for transfer in reinforcement learning","authors":"André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, David Silver","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barreto et al., 2019] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel K Toyama, Jonathan J Hunt, Shibl Mourad, David Silver, et al. The option keyboard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems, 2019.","title":"The option keyboard: Combining skills in reinforcement learning","authors":"André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Jonathan J Daniel K Toyama, Shibl Hunt, David Mourad, Silver","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dayan, 1993] Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-624, 1993.","title":"Improving generalization for temporal difference learning: The successor representation","authors":"Peter Dayan","year":"1993","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dean and Givan, 1997] Thomas Dean and Robert Givan. Model minimization in markov decision processes. In AAAI Conference on Artificial Intelligence, 1997.","title":"Model minimization in markov decision processes","authors":"Thomas Dean, Robert Givan","year":"1997","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ferns et al., 2004] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision pro- cesses. In Uncertainty in Artifical Intelligence, 2004.","title":"Metrics for finite markov decision processes","authors":"Ferns","year":"2004","venue":"Uncertainty in Artifical Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In International Conference on Computer Vision, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2016] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In International Joint Conference on Artificial Intelligence, 2016.","title":"The malmo platform for artificial intelligence experimentation","authors":"Matthew Johnson, Katja Hofmann, Tim Hutton, David Bignell","year":"2016","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/643"},{"position":12,"raw":"[Konidaris and Barto, 2007] George Dimitri Konidaris and Andrew G Barto. Building portable options: Skill transfer in reinforcement learning. In International Joint Confer- ence on Artificial Intelligence, pages 895-900, 2007.","title":"Building portable options: Skill transfer in reinforcement learning","authors":"Barto ; Konidaris, Dimitri George, Andrew G Konidaris, Barto","year":"2007","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2007/144"},{"position":13,"raw":"[Li et al., 2006] Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. In International Symposium on Artificial Intelli- gence and Mathematics, 2006.","title":"Towards a unified theory of state abstraction for mdps","authors":"Lihong Li, Thomas J Walsh, Michael L Littman","year":"2006","venue":"International Symposium on Artificial Intelligence and Mathematics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Machado et al., 2018] Marlos C Machado, Clemens Rosen- baum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Mur- ray Campbell. Eigenoption discovery through the deep successor representation. In International Conference on Learning Representations, 2018.","title":"Eigenoption discovery through the deep successor representation","authors":"C Marlos, Clemens Machado, Xiaoxiao Rosenbaum, Miao Guo, Gerald Liu, Murray Tesauro, Campbell","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mehta et al., 2008] Neville Mehta, Sriraam Natarajan, Prasad Tadepalli, and Alan Fern. Transfer in variable- reward hierarchical reinforcement learning. Machine Learning, 73(3):289, 2008.","title":"Transfer in variablereward hierarchical reinforcement learning","authors":"Neville Mehta, Sriraam Natarajan, Prasad Tadepalli, Alan Fern","year":"2008","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pons and Latapy, 2005] Pascal Pons and Matthieu Latapy. Computing communities in large networks using random walks. In International Symposium on Computer and In- formation Sciences, pages 284-293. Springer, 2005.","title":"Computing communities in large networks using random walks","authors":"Latapy Pons, Pascal Pons, Matthieu Latapy","year":"2005","venue":"International Symposium on Computer and Information Sciences","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ramesh et al., 2019] Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option dis- covery framework for reinforcement learning. In Interna- tional Joint Conference on Artificial Intelligence, 2019.","title":"Successor options: An option discovery framework for reinforcement learning","authors":"Rahul Ramesh, Manan Tomar, Balaraman Ravindran","year":"2019","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/458"},{"position":19,"raw":"[Ravindran and Barto, 2002] Balaraman Ravindran and An- drew G Barto. Model minimization in hierarchical rein- forcement learning. In International Symposium on Ab- straction, Reformulation, and Approximation, pages 196- 211. Springer, 2002.","title":"Model minimization in hierarchical reinforcement learning","authors":"Barto Ravindran, Balaraman Ravindran, Andrew G Barto","year":"2002","venue":"International Symposium on Abstraction, Reformulation, and Approximation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ravindran and Barto, 2003] Balaraman Ravindran and An- drew G Barto. Smdp homomorphisms: an algebraic ap- proach to abstraction in semi-markov decision processes. In International Joint Conference on Artificial Intelli- gence, pages 1011-1016, 2003.","title":"Smdp homomorphisms: an algebraic approach to abstraction in semi-markov decision processes","authors":"Barto Ravindran, Balaraman Ravindran, Andrew G Barto","year":"2003","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2003/144"},{"position":21,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Syed et al., 2008] Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In International Conference on Machine Learning, pages 1032-1039, 2008.","title":"Apprenticeship learning using linear programming","authors":"Syed","year":"2008","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}