{"type":"ScholarlyArticle","title":"Guide to Control: Offline Hierarchical Reinforcement Learning Using Subgoal Generation for Long-Horizon and Sparse-Reward Tasks","authors":[{"name":"Wonchul Shin","slug":"wonchul-shin","given":"Wonchul","family":"Shin","affiliation":"Sungkyunkwan University"},{"name":"Yusung Kim","slug":"yusung-kim","given":"Yusung","family":"Kim","affiliation":"Sungkyunkwan University"}],"abstract":"Reinforcement learning (RL) has achieved considerable success in many fields, but applying it to real-world problems can be costly and risky because it requires a lot of online interaction. Recently, offline RL has shown the possibility of extracting a solution through existing logged data without online interaction. In this work, we propose an offline hierarchical RL method, Guider (Guide to Control), that can efficiently solve long-horizon and sparse-reward tasks from offline data. The high-level policy sequentially generates a subgoal that can guide the agent to arrive at the final goal, and the lower-level policy learns how to reach each given guided subgoal. In the process of learning from offline data, the key is to make the low-level policy reachable to the generated subgoals. We show that high-quality subgoal generation is possible through pre-training a latent subgoal prior model. The well-regulated subgoal generation improves performance while avoiding distributional shifts in offline RL by breaking down long, complex tasks into shorter, easier ones. For evaluations, Guider outperforms prior offline RL methods in long-horizon robot navigation and complex manipulation benchmarks. Our code is available at https://github.com/gckor/Guider.","doi":"10.24963/ijcai.2023/469","url":"https://ijcai.org/proceedings/2023/469","pdf":"https://ijcai.org/proceedings/2023/0469.pdf","markdown":"https://ijcai.org/proceedings/2023/469.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/469","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":469,"pages":{"first":4217,"last":4225},"keywords":["Robotics: ROB: Behavior and control","Machine Learning: ML: Deep reinforcement learning","Planning and Scheduling: PS: Learning in planning and scheduling"],"videos":[],"references":[{"position":1,"raw":"[Ajay et al., 2021] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Repre- sentations, 2021.","title":"Opal: Offline primitive discovery for accelerating offline reinforcement learning","authors":"Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, Ofir Nachum","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Ad- vances in Neural Information Processing Systems, pages 5048--5058, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mcgrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bagaria et al., 2021] Akhil Bagaria, Jason Senthil, Matthew Slivinski, and George Konidaris. Robustly learning com- posable options in deep reinforcement learning. In Inter- national Joint Conference on Artificial Intelligence, 2021.","title":"Robustly learning composable options in deep reinforcement learning","authors":"Akhil Bagaria, Jason Senthil, Matthew Slivinski, George Konidaris","year":"2021","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/298"},{"position":4,"raw":"[Balaji et al., 2019] Bharathan Balaji, Sunil Mallya, Sahika Genc, Saurabh Gupta, Leo Dirac, Vineet Khare, Gourav Roy, Tao Sun, Yunzhe Tao, Brian Townsend, Ed- die Calleja, Sunil Muralidhara, and Dhanasekar Karup- pasamy. Deepracer: Educational autonomous racing plat- form for experimentation with sim2real reinforcement learning. arXiv preprint arXiv:1911.01562, 2019.","title":"Deepracer: Educational autonomous racing platform for experimentation with sim2real reinforcement learning","authors":"Bharathan Balaji, Sunil Mallya, Sahika Genc, Saurabh Gupta, Leo Dirac, Vineet Khare, Gourav Roy, Tao Sun, Yunzhe Tao, Brian Townsend, Eddie Calleja, Sunil Muralidhara, Dhanasekar Karuppasamy","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Ma- chine Learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghosh et al., 2021] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated super- vised learning. In International Conference on Learning Representations, 2021.","title":"Learning to reach goals via iterated supervised learning","authors":"Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, Sergey Levine","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learn- ing with deep energy-based policies. In International Con- ference on Machine Learning, pages 1352-1361. PMLR, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hafner et al., 2020] Danijar Hafner, Timothy Lillicrap, Mo- hammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.","title":"Mastering atari with discrete world models","authors":"Hafner","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hafner et al., 2022] Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. arXiv preprint arXiv:2206.04114, 2022.","title":"Deep hierarchical planning from pixels","authors":"Hafner","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kalashnikov et al., 2018] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vin- cent Vanhoucke, and Sergey Levine. Scalable deep rein- forcement learning for vision-based robotic manipulation. In Conference on Robot Learning, 2018.","title":"Scalable deep reinforcement learning for vision-based robotic manipulation","authors":"Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, Sergey Levine","year":"2018","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kostrikov et al., 2021] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with im- plicit q-learning. arXiv preprint arXiv:2110.06169, 2021.","title":"Offline reinforcement learning with implicit q-learning","authors":"Kostrikov","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Servey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Servey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levy et al., 2019] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations, 2019.","title":"Learning multi-level hierarchies with hindsight","authors":"Andrew Levy, George Konidaris, Robert Platt, Kate Saenko","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021] Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. Learning subgoal representations with slow dynamics. In International Conference on Learning Representations, 2021.","title":"Learning subgoal representations with slow dynamics","authors":"Siyuan Li, Lulu Zheng, Jianhao Wang, Chongjie Zhang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lynch et al., 2020] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonatah Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Con- ference on Robot Learning, 2020.","title":"Learning latent plans from play","authors":"Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonatah Tompson, Sergey Levine, Pierre Sermanet","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nachum et al., 2018] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical rein- forcement learning. In Advances in Neural Information Processing Systems, pages 3307-3317, 2018.","title":"Data-efficient hierarchical reinforcement learning","authors":"Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nachum et al., 2019] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019.","title":"Algaedice: Policy gradient from arbitrary experience","authors":"Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, Dale Schuurmans","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nair et al., 2020] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online re- inforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.","title":"Awac: Accelerating online reinforcement learning with offline datasets","authors":"Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peng et al., 2019] Xue B. Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learn- ing. arXiv preprint arXiv:1910.00177, 2019.","title":"Advantage-weighted regression: Simple and scalable off-policy reinforcement learning","authors":"B Xue, Aviral Peng, Grace Kumar, Sergey Zhang, Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rafailov et al., 2020] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. arXiv preprint arXiv:2012.11547, 2020.","title":"Offline reinforcement learning from images with latent space models","authors":"Rafailov","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Mad- dison, Arthur Guez, Laurent Sifre, George van den Driess- che1, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Driessche1, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van ; Marc, Sander Lanctot, Dominik Dieleman, John Grewe, Nal Nham, Ilya Kalchbrenner, Timothy Sutskever, Madeleine Lillicrap, Koray Leach, Thore Kavukcuoglu, Demis Graepel, Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2022] Vivienne H. Wang, Joni Pajarinen, Tinghuai Wang, and Joni-Kristian Kamarainen. Hierarchi- cal reinforcement learning with adversarially guided sub- goals. arXiv preprint arXiv:2201.09635, 2022.","title":"Hierarchical reinforcement learning with adversarially guided subgoals","authors":"Vivienne H Wang, Joni Pajarinen, Tinghuai Wang, Joni-Kristian Kamarainen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2019] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.","title":"Behavior regularized offline reinforcement learning","authors":"Yifan Wu, George Tucker, Ofir Nachum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2022] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. In International Confer- ence on Learning Representations, 2022.","title":"Rethinking goal-conditioned supervised learning and its connection to offline rl","authors":"Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, Chongjie Zhang","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrin- sic options. In International Conference on Learning Rep- resentations, 2020.","title":"Hierarchical reinforcement learning by discovering intrinsic options","authors":"Jesse Zhang, Haonan Yu, Wei Xu","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}