{"type":"ScholarlyArticle","title":"Efficient Offline Meta-Reinforcement Learning via Robust Task Representations and Adaptive Policy Generation","authors":[{"name":"Zhengwei Li","slug":"zhengwei-li","given":"Zhengwei","family":"Li","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Science (UCAS) , 100049 , Beijing , China; National Key Laboratory of Multimodal Artificial Intelligence Systems , Institute of Automation , Chinese Academy of Sciences , Beijing , 100190 , China"},{"name":"Zhenyang Lin","slug":"zhenyang-lin","given":"Zhenyang","family":"Lin","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Science (UCAS) , 100049 , Beijing , China; National Key Laboratory of Multimodal Artificial Intelligence Systems , Institute of Automation , Chinese Academy of Sciences , Beijing , 100190 , China"},{"name":"Yurou Chen","slug":"yurou-chen","given":"Yurou","family":"Chen","affiliation":"National Key Laboratory of Multimodal Artificial Intelligence Systems , Institute of Automation , Chinese Academy of Sciences , Beijing , 100190 , China"},{"name":"Zhiyong Liu","slug":"zhiyong-liu","given":"Zhiyong","family":"Liu","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Science (UCAS) , 100049 , Beijing , China; National Key Laboratory of Multimodal Artificial Intelligence Systems , Institute of Automation , Chinese Academy of Sciences , Beijing , 100190 , China"}],"abstract":"Zero-shot adaptation is crucial for agents facing new tasks. Offline Meta-Reinforcement Learning (OMRL), utilizing offline multi-task datasets to train policies, offers a way to attain this ability. Although most OMRL methods construct task representations via contrastive learning and merge them with states for policy input, these methods may have inherent problems. Specifically, integrating task representations with states for policy input limits learning efficiency, due to failing to leverage the similarities among tasks. Moreover, uniformly sampling an equal number of negative samples from different tasks in contrastive learning can hinder differentiation of more similar tasks, potentially diminishing task representation robustness. In this paper, we introduce an OMRL algorithm to tackle the aforementioned issues. We design a network structure for efficient learning by leveraging task similarity. It features shared lower layers for common feature extraction with a hypernetworks-driven upper layer, customized to process features per task's attributes. Furthermore, to achieve robust task representations for generating task-specific control policies, we utilize contrastive learning and introduce a novel method to construct negative sample pairs based on task similarity. Experimental results show that our method notably boosts learning efficiency and zero-shot adaptation in new tasks, surpassing previous methods across multiple challenging domains.","doi":"10.24963/ijcai.2024/500","url":"https://ijcai.org/proceedings/2024/500","pdf":"https://ijcai.org/proceedings/2024/0500.pdf","markdown":"https://ijcai.org/proceedings/2024/500.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/500","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":500,"pages":{"first":4524,"last":4532},"keywords":["Machine Learning: ML: Reinforcement learning","Robotics: ROB: Learning in robotics","Machine Learning: ML: Offline reinforcement learning","Machine Learning: ML: Meta-learning"],"videos":[],"references":[{"position":1,"raw":"[Ball et al., 2021] Philip J Ball, Cong Lu, Jack Parker- Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In International Conference on Ma- chine Learning, pages 619-629. PMLR, 2021.","title":"Augmented world models facilitate zero-shot dynamics generalization from a single offline environment","authors":"J Philip, Cong Ball, Jack Lu, Stephen Parker-Holder, Roberts","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beck et al., 2023a] Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta-reinforcement learning. In Conference on Robot Learning, pages 1478-1487. PMLR, 2023.","title":"Hypernetworks in meta-reinforcement learning","authors":"Beck","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beck et al., 2023b] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shi- mon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023.","title":"A survey of meta-reinforcement learning","authors":"Beck","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellman, 1966] Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.","title":"Dynamic programming","authors":"Richard Bellman","year":"1966","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Beukman et al., 2023] Michael Beukman, Devon Jarvis, Richard Klein, Steven James, and Benjamin Ros- man. Dynamics generalisation in reinforcement learn- ing via adaptive context-aware policies. arXiv preprint arXiv:2310.16686, 2023.","title":"Dynamics generalisation in reinforcement learning via adaptive context-aware policies","authors":"Michael Beukman, Devon Jarvis, Richard Klein, Steven James, Benjamin Rosman","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dorfman et al., 2021] Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning- identifiability challenges and effective data collection strategies. Advances in Neural Information Processing Systems, 34:4607-4618, 2021.","title":"Offline meta reinforcement learningidentifiability challenges and effective data collection strategies","authors":"Ron Dorfman, Idan Shenfeld, Aviv Tamar","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2016] Yan Duan, John Schulman, Xi Chen, Pe- ter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learn- ing. arXiv preprint arXiv:1611.02779, 2016.","title":"Rl 2 : Fast reinforcement learning via slow reinforcement learning","authors":"Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Duff, 2002] Michael O'Gordon Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov de- cision processes. University of Massachusetts Amherst, 2002.","title":"Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes","authors":"O'gordon Michael, Duff","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fujimoto and Gu, 2021] Scott Fujimoto and Shixi- ang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.","title":"A minimalist approach to offline reinforcement learning","authors":"Gu Fujimoto, Scott Fujimoto, Shixiang Shane Gu","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on ma- chine learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gao et al., 2023] Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu, Qi Yi, Shaohui Peng, Siming Lan, Ruizhi Chen, Zidong Du, Xing Hu, et al. Context shift reduction for offline meta-reinforcement learning. arXiv preprint arXiv:2311.03695, 2023.","title":"Context shift reduction for offline meta-reinforcement learning","authors":"Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu, Qi Yi, Shaohui Peng, Siming Lan, Ruizhi Chen, Zidong Du, Xing Hu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ha et al., 2016] David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks, 2016. [Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"David Ha, Andrew Dai, V Quoc, Le, Hypernetworks, Haarnoja","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hessel et al., 2019] Matteo Hessel, Hubert Soyer, Lasse Es- peholt, Wojciech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 33, pages 3796-3803, 2019.","title":"Multi-task deep reinforcement learning with popart","authors":"Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, Hado Van Hasselt","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kumar et al., 2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.","title":"Stabilizing off-policy q-learning via bootstrapping error reduction","authors":"Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Infor- mation Processing Systems, 33:1179-1191, 2020.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lange et al., 2012] Sascha Lange, Thomas Gabel, and Mar- tin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pages 45-73. Springer, 2012.","title":"Batch reinforcement learning","authors":"Sascha Lange, Thomas Gabel, Martin Riedmiller","year":"2012","venue":"Reinforcement learning: State-of-the-art","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2020] Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. Advances in Neural Information Processing Sys- tems, 33:6197-6210, 2020.","title":"Multi-task batch reinforcement learning with metric learning","authors":"Li","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2021] Lanqing Li, Rui Yang, and Dijun Luo. FO- CAL: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization. In International Conference on Learning Representations, 2021.","title":"FO-CAL: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization","authors":"Lanqing Li, Rui Yang, Dijun Luo","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pong et al., 2022] Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta-reinforcement learning with online self-supervision. In International Conference on Machine Learning, pages 17811-17829. PMLR, 2022.","title":"Offline meta-reinforcement learning with online self-supervision","authors":"Ashvin V Vitchyr H Pong, Laura M Nair, Catherine Smith, Sergey Huang, Levine","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rakelly et al., 2019a] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinforcement learning via probabilistic con- text variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.","title":"Efficient offpolicy meta-reinforcement learning via probabilistic context variables","authors":"Rakelly","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rakelly et al., 2019b] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinforcement learning via probabilistic con- text variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.","title":"Efficient offpolicy meta-reinforcement learning via probabilistic context variables","authors":"Rakelly","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sarafian et al., 2021] Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In International Conference on Machine Learning, pages 9301-9312.","title":"Recomposing the reinforcement learning building blocks with hypernetworks","authors":"Elad Sarafian, Shai Keynan, Sarit Kraus","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shinzaki et al., 2021] Masao Shinzaki, Yusuke Koda, Koji Yamamoto, Takayuki Nishio, Masahiro Morikura, Yushi Shirato, Daisei Uchida, and Naoki Kita. Zero-shot adap- tation for mmwave beam-tracking on overhead messen- ger wires through robust adversarial reinforcement learn- ing. IEEE Transactions on Cognitive Communications and Networking, 8(1):232-245, 2021.","title":"Zero-shot adaptation for mmwave beam-tracking on overhead messenger wires through robust adversarial reinforcement learning","authors":"Masao Shinzaki, Yusuke Koda, Koji Yamamoto, Takayuki Nishio, Masahiro Morikura, Yushi Shirato, Daisei Uchida, Naoki Kita","year":"2021","venue":"IEEE Transactions on Cognitive Communications and Networking","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sohn et al., 2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural infor- mation processing systems, 28, 2015.","title":"Learning structured output representation using deep conditional generative models","authors":"Sohn","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2016] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016.","title":"Learning to reinforcement learn","authors":"Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, Matt Botvinick","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2023a] Jianhao Wang, Jin Zhang, Haozhe Jiang, Junyu Zhang, Liwei Wang, and Chongjie Zhang. Offline meta reinforcement learning with in-distribution online adaptation. arXiv preprint arXiv:2305.19529, 2023.","title":"Offline meta reinforcement learning with in-distribution online adaptation","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2023b] Mingyang Wang, Zhenshan Bing, Xi- angtong Yao, Shuai Wang, Huang Kai, Hang Su, Chen- guang Yang, and Alois Knoll. Meta-reinforcement learn- ing based on self-supervised task representation learning. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 37, pages 10157-10165, 2023.","title":"Meta-reinforcement learning based on self-supervised task representation learning","authors":"Wang","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2023] Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper-decision transformer for efficient online policy adaptation. arXiv preprint arXiv:2304.08487, 2023.","title":"Hyper-decision transformer for efficient online policy adaptation","authors":"Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yuan and Lu, 2022] Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinforcement learn- ing via contrastive learning. In International Conference on Machine Learning, pages 25747-25759. PMLR, 2022.","title":"Robust task representations for offline meta-reinforcement learning via contrastive learning","authors":"Lu ; Yuan, Haoqi Yuan, Zongqing Lu","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang and Yang, 2021] Yu Zhang and Qiang Yang. A sur- vey on multi-task learning. IEEE Transactions on Knowl- edge and Data Engineering, 34(12):5586-5609, 2021.","title":"A survey on multi-task learning","authors":"Yu Zhang, Qiang Yang","year":"2021","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zintgraf et al., 2020] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hof- mann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representation (ICLR), 2020.","title":"Varibad: A very good method for bayes-adaptive deep rl via meta-learning","authors":"Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson","year":"2020","venue":"International Conference on Learning Representation (ICLR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}