{"type":"ScholarlyArticle","title":"Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Tasks","authors":[{"name":"Donghoon Kim","slug":"donghoon-kim","given":"Donghoon","family":"Kim","affiliation":"Department of Artificial Intelligence , Sungkyunkwan University"},{"name":"Minjong Yoo","slug":"minjong-yoo","given":"Minjong","family":"Yoo","affiliation":"Department of Computer Science and Engineering , Sungkyunkwan University"},{"name":"Honguk Woo","slug":"honguk-woo","given":"Honguk","family":"Woo","affiliation":"Department of Artificial Intelligence , Sungkyunkwan University; Department of Computer Science and Engineering , Sungkyunkwan University"}],"abstract":"Goal-conditioned (GC) policy learning often faces a challenge arising from the sparsity of rewards, when confronting long-horizon goals. To address the challenge, we explore skill-based GC policy learning in offline settings, where skills are acquired from existing data and long-horizon goals are decomposed into sequences of near-term goals that align with these skills. Specifically, we present an `offline GC policy learning via skill-step abstraction' framework (GLvSA) tailored for tackling long-horizon GC tasks affected by goal distribution shifts. In the framework, a GC policy is progressively learned offline in conjunction with the incremental modeling of skill-step abstractions on the data. We also devise a GC policy hierarchy that not only accelerates GC policy learning within the framework but also allows for parameter-efficient fine-tuning of the policy. Through experiments with the maze and Franka kitchen environments, we demonstrate the superiority and efficiency of our GLvSA framework in adapting GC policies to a wide range of long-horizon goals. The framework achieves competitive zero-shot and few-shot adaptation performance, outperforming existing GC policy learning and skill-based methods.","doi":"10.24963/ijcai.2024/473","url":"https://ijcai.org/proceedings/2024/473","pdf":"https://ijcai.org/proceedings/2024/0473.pdf","markdown":"https://ijcai.org/proceedings/2024/473.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/473","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":473,"pages":{"first":4282,"last":4290},"keywords":["Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Chebotar et al., 2021] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforce- ment learning of robotic skills. In Proc.of the 38th Inter- national Conference on Machine Learning (ICML 2021), pages 1518-1528. PMLR, 2021.","title":"Actionable models: Unsupervised offline reinforcement learning of robotic skills","authors":"Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn","year":"2021","venue":"Proc.of the 38th International Conference on Machine Learning (ICML 2021)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2023] Daesol Cho, Seungjae Lee, and H. Jin Kim. Outcome-directed reinforcement learning by uncer- tainty \\& temporal distance-aware curriculum goal gen- eration. In Proc.of the 11th International Conference on Learning Representations (ICLR 2023), 2023.","title":"Outcome-directed reinforcement learning by uncertainty \\& temporal distance-aware curriculum goal generation","authors":"Cho","year":"2023","venue":"Proc.of the 11th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Egorov and Shpilman, 2022] Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforce- ment learning. In Proc.of the 21st International Confer- ence on Autonomous Agents and Multiagent Systems (AA- MAS 2022), pages 381-390, 2022.","title":"Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforcement learning","authors":"Shpilman Egorov","year":"2022","venue":"Proc.of the 21st International Conference on Autonomous Agents and Multiagent Systems (AA-MAS 2022)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2022] Kuan Fang, Patrick Yin, Ashvin Nair, and Sergey Levine. Planning to practice: Efficient online fine- tuning by composing goals in latent space. In Proc.of the 35th IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS 2022), pages 4076-4083. IEEE, 2022.","title":"Planning to practice: Efficient online finetuning by composing goals in latent space","authors":"Kuan Fang, Patrick Yin, Ashvin Nair, Sergey Levine","year":"2022","venue":"Proc.of the 35th IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghosh et al., 2021] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysen- bach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In Proc of the 9th Inter- national Conference on Learning Representations (ICLR 2021), 2021.","title":"Learning to reach goals via iterated supervised learning","authors":"Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, Sergey Levine","year":"2021","venue":"Proc of the 9th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learn- ing with a stochastic actor. In Proc.of the 35th Inter- national conference on machine learning (ICML 2018), pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"Proc.of the 35th International conference on machine learning (ICML 2018)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hafner et al., 2019] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.","title":"Dream to control: Learning behaviors by latent imagination","authors":"Hafner","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hafner et al., 2020] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In Proc.of the 8th International Conference on Learning Representations (ICLR 2020), 2020.","title":"Mastering atari with discrete world models","authors":"Hafner","year":"2020","venue":"Proc.of the 8th International Conference on Learning Representations (ICLR 2020)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hansen et al., 2022] Nicklas A Hansen, Hao Su, and Xiao- long Wang. Temporal difference learning for model pre- dictive control. In Proc.of the 39th International Con- ference on Machine Learning (ICML 2022), pages 8387- 8406. PMLR, 2022.","title":"Temporal difference learning for model predictive control","authors":"Hao Nicklas A Hansen, Xiaolong Su, Wang","year":"2022","venue":"Proc.of the 39th International Conference on Machine Learning (ICML 2022)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Higgins et al., 2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained varia- tional framework. In Proc.of the 5th International confer- ence on learning representations (ICLR 2017), 2017.","title":"beta-vae: Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner","year":"2017","venue":"Proc.of the 5th International conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2023] Edward S. Hu, Richard Chang, Oleh Ry- bkin, and Dinesh Jayaraman. Planning goals for explo- ration. In Proc.of the 11th International Conference on Learning Representations (ICLR 2023), 2023.","title":"Planning goals for exploration","authors":"Edward S Hu, Richard Chang, Oleh Rybkin, Dinesh Jayaraman","year":"2023","venue":"Proc.of the 11th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hussonnois et al., 2023] Maxence Hussonnois, Thom- men George Karimpanal, and Santu Rana. Controlled diversity with preference: Towards learning a diverse set of desired skills. In Proc.of the 22nd International Con- ference on Autonomous Agents and Multiagent Systems (AAMAS 2023), pages 1135-1143, 2023.","title":"Maxence Hussonnois, Thommen George Karimpanal, and Santu Rana. Controlled diversity with preference: Towards learning a diverse set of desired skills","authors":"Hussonnois","year":"2023","venue":"Proc.of the 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2023)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Janner et al., 2019] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Proc.of the 33rd Ad- vances in neural information processing systems (NeurIPS 2019), volume 32, 2019.","title":"When to trust your model: Model-based policy optimization","authors":"Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine","year":"2019","venue":"Proc.of the 33rd Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiang et al., 2022] Yiding Jiang, Evan Liu, Benjamin Ey- senbach, J Zico Kolter, and Chelsea Finn. Learning op- tions via compression. In Proc.of the 36th Advances in Neural Information Processing Systems (NeurIPS 2022), volume 35, pages 21184-21199, 2022.","title":"Learning options via compression","authors":"Jiang","year":"2022","venue":"Proc.of the 36th Advances in Neural Information Processing Systems (NeurIPS 2022)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2023] Junsu Kim, Younggyo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating graph-based planning with goal-conditioned policies. In Proc.of the 11th International Conference on Learning Representa- tions (ICLR 2023), 2023.","title":"Imitating graph-based planning with goal-conditioned policies","authors":"Kim","year":"2023","venue":"Proc.of the 11th International Conference on Learning Representations (ICLR 2023)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2022] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.08299, 2022.","title":"Goal-conditioned reinforcement learning: Problems and solutions","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/770"},{"position":18,"raw":"[Lu et al., 2020] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill- space planning. In Proc.of the 8th International Confer- ence on Learning Representations (ICLR 2020), 2020.","title":"Reset-free lifelong learning with skillspace planning","authors":"Kevin Lu, Aditya Grover, Pieter Abbeel, Igor Mordatch","year":"2020","venue":"Proc.of the 8th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2022] Jason Yecheng Ma, Jason Yan, Dinesh Ja- yaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f -advantage regression. In Proc.of the 36th Advances in Neural Information Process- ing Systems (NeurIPS 2022), volume 35, pages 310-323, 2022.","title":"Offline goal-conditioned reinforcement learning via f -advantage regression","authors":"Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, Osbert Bastani","year":"2022","venue":"Proc.of the 36th Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mendonca et al., 2021] Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. In Proc.of the 35th Advances in Neural Information Process- ing Systems (NeurIPS 2021), volume 34, pages 24379- 24391, 2021.","title":"Discovering and achieving goals via world models","authors":"Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, Deepak Pathak","year":"2021","venue":"Proc.of the 35th Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nair et al., 2020] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online re- inforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.","title":"Awac: Accelerating online reinforcement learning with offline datasets","authors":"Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nam et al., 2022] Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph Jaewhan Lim. Skill- based meta-reinforcement learning. In Proc.of the 10th International Conference on Learning Representations (ICLR 2022), 2022.","title":"Skillbased meta-reinforcement learning","authors":"Nam","year":"2022","venue":"Proc.of the 10th International Conference on Learning Representations (ICLR 2022)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pertsch et al., 2021] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Proc.of the 5th Conference on robot learning (CoRL 2021), pages 188-204. PMLR, 2021.","title":"Accelerating reinforcement learning with learned skill priors","authors":"Karl Pertsch, Youngwoon Lee, Joseph Lim","year":"2021","venue":"Proc.of the 5th Conference on robot learning (CoRL 2021)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pertsch et al., 2022] Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J Lim. Guided reinforcement learning with learned skills. In Proc.of the 6th Conference on Robot Learning (CoRL 2022), pages 729-739. PMLR, 2022.","title":"Guided reinforcement learning with learned skills","authors":"Karl Pertsch, Youngwoon Lee, Yue Wu, Joseph J Lim","year":"2022","venue":"Proc.of the 6th Conference on Robot Learning (CoRL 2022)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pertsch et al., 2023] Karl Pertsch, Ruta Desai, Vikash Ku- mar, Franziska Meier, Joseph J Lim, Dhruv Batra, and Ak- shara Rai. Cross-domain transfer via semantic skill imita- tion. In Proc.of the 7th Conference on Robot Learning (CoRL 2023), pages 690-700. PMLR, 2023.","title":"Cross-domain transfer via semantic skill imitation","authors":"Karl Pertsch, Ruta Desai, Vikash Kumar, Franziska Meier, Joseph J Lim, Dhruv Batra, Akshara Rai","year":"2023","venue":"Proc.of the 7th Conference on Robot Learning (CoRL 2023)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pitis et al., 2022] Silviu Pitis, Elliot Creager, Ajay Man- dlekar, and Animesh Garg. Mocoda: Model-based coun- terfactual data augmentation. In Proc.of the 36th Advances in Neural Information Processing Systems (NeurIPS 2022), volume 35, pages 18143-18156, 2022.","title":"Mocoda: Model-based counterfactual data augmentation","authors":"Silviu Pitis, Elliot Creager, Ajay Mandlekar, Animesh Garg","year":"2022","venue":"Proc.of the 36th Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shi et al., 2023] Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. In Proc.of the 7th Conference on Robot Learning (CoRL 2023), pages 2262-2272. PMLR, 2023.","title":"Skill-based model-based reinforcement learning","authors":"Lucy Xiaoyang Shi, Joseph J Lim, Youngwoon Lee","year":"2023","venue":"Proc.of the 7th Conference on Robot Learning (CoRL 2023)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sohn et al., 2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Proc.of the 29th Ad- vances in neural information processing systems (NeurIPS 2015), volume 28, 2015.","title":"Learning structured output representation using deep conditional generative models","authors":"Sohn","year":"2015","venue":"Proc.of the 29th Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2021] Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. In Proc.of the 35th Advances in Neural Infor- mation Processing Systems (NeurIPS 2021), volume 34, pages 29420-29432, 2021.","title":"Offline reinforcement learning with reverse model-based imagination","authors":"Wang","year":"2021","venue":"Proc.of the 35th Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2023] Caroline Wang, Garrett Warnell, and Pe- ter Stone. D-shape: Demonstration-shaped reinforcement learning via goal-conditioning. In Proc.of the 22nd Inter- national Conference on Autonomous Agents and Multia- gent Systems (AAMAS 2023), pages 1267-1275, 2023.","title":"D-shape: Demonstration-shaped reinforcement learning via goal-conditioning","authors":"Caroline Wang, Garrett Warnell, Peter Stone","year":"2023","venue":"Proc.of the 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2023)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2021] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. In Proc.of the 9th Inter- national Conference on Learning Representations (ICLR 2021), 2021.","title":"Rethinking goal-conditioned supervised learning and its connection to offline rl","authors":"Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, Chongjie Zhang","year":"2021","venue":"Proc.of the 9th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}