{"type":"ScholarlyArticle","title":"ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles","authors":[{"name":"Kai Zhao","slug":"kai-zhao","given":"Kai","family":"Zhao","affiliation":"College of Intelligence and Computing , Tianjin University; Bilibili"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yi Ma","slug":"yi-ma","given":"Yi","family":"Ma","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Jinyi Liu","slug":"jinyi-liu","given":"Jinyi","family":"Liu","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yan Zheng","slug":"yan-zheng","given":"Yan","family":"Zheng","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Zhaopeng Meng","slug":"zhaopeng-meng","given":"Zhaopeng","family":"Meng","affiliation":"College of Intelligence and Computing , Tianjin University"}],"abstract":"Offline reinforcement learning (RL) is a learning paradigm where an agent learns from a fixed dataset of experience. However, learning solely from a static dataset can limit the performance due to the lack of exploration. To overcome it, offline-to-online RL combines offline pre-training with online fine-tuning, which enables the agent to further refine its policy by interacting with the environment in real-time. Despite its benefits, existing offline-to-online RL methods suffer from performance degradation and slow improvement during the online phase. To tackle these challenges, we propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Moreover, to expedite online performance enhancement, we appropriately loosen the pessimism of Q-value estimation and incorporate ensemble-based exploration mechanisms into our framework. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods during online fine-tuning on a range of locomotion and navigation tasks, significantly outperforming existing offline-to-online RL methods.","doi":"10.24963/ijcai.2024/615","url":"https://ijcai.org/proceedings/2024/615","pdf":"https://ijcai.org/proceedings/2024/0615.pdf","markdown":"https://ijcai.org/proceedings/2024/615.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/615","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":615,"pages":{"first":5563,"last":5571},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Ensemble methods","Machine Learning: ML: Offline reinforcement learning","Machine Learning: ML: Online learning"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2020] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on of- fline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.","title":"An optimistic perspective on offline reinforcement learning","authors":"Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An et al., 2021] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline rein- forcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.","title":"Uncertainty-based offline reinforcement learning with diversified q-ensemble","authors":"Seungyong Gaon An, Jang-Hyun Moon, Hyun Oh Kim, Song","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2022] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning. arXiv preprint arXiv:2202.11566, 2022.","title":"Pessimistic bootstrapping for uncertaintydriven offline reinforcement learning","authors":"Bai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q- learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.","title":"Randomized ensembled double qlearning: Learning fast without a model","authors":"Xinyue Chen, Che Wang, Zijian Zhou, Keith Ross","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage- weighted policy optimization for offline rl. arXiv preprint arXiv:2203.08949, 2022.","title":"Latent-variable advantageweighted policy optimization for offline rl","authors":"Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, Chongjie Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ciosek et al., 2019] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with op- timistic actor critic. Advances in Neural Information Pro- cessing Systems, 32, 2019.","title":"Better exploration with optimistic actor critic","authors":"Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ecoffet et al., 2021] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-586, 2021.","title":"First return, then explore","authors":"Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fort et al., 2019] Stanislav Fort, Huiyi Hu, and Balaji Lak- shminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv:1912.02757, 2019.","title":"Deep ensembles: A loss landscape perspective","authors":"Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fujimoto and Gu, 2021] Scott Fujimoto and Shixi- ang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.","title":"A minimalist approach to offline reinforcement learning","authors":"Gu Fujimoto, Scott Fujimoto, Shixiang Shane Gu","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on ma- chine learning, pages 2052-2062. PMLR, 2019. [Ghasemipour et al., 2022]","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. arXiv preprint arXiv:2205.13703, 2022.","title":"Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters","authors":"Seyed Kamyar, Seyed Ghasemipour, Shixiang Shane Gu, Ofir Nachum","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hao et al., 2023] Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Transac- tions on Neural Networks and Learning Systems, pages 1- 21, 2023.","title":"Exploration in deep reinforcement learning: From single-agent to multiagent domain","authors":"Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, Zhen Wang","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.","title":"Vime: Variational information maximizing exploration","authors":"Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kidambi et al., 2020] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21823, 2020.","title":"Morel: Model-based offline reinforcement learning","authors":"Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kostrikov et al., 2021] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with im- plicit q-learning. arXiv preprint arXiv:2110.06169, 2021.","title":"Offline reinforcement learning with implicit q-learning","authors":"Kostrikov","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Infor- mation Processing Systems, 33:1179-1191, 2020.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lee et al., 2021] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learn- ing, pages 6131-6141. PMLR, 2021.","title":"Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning","authors":"Lee","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lee et al., 2022] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online re- inforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.","title":"Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble","authors":"Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, Jinwoo Shin","year":"2022","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2023] Jinyi Liu, Yi Ma, Jianye Hao, Yujing Hu, Yan Zheng, Tangjie Lv, and Changjie Fan. Prioritized tra- jectory replay: A replay memory for data-driven reinforce- ment learning. CoRR, abs/2306.15503, 2023.","title":"Prioritized trajectory replay: A replay memory for data-driven reinforcement learning","authors":"Jinyi Liu, Yi Ma, Jianye Hao, Yujing Hu, Yan Zheng, Tangjie Lv, Changjie Fan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2024] Jinyi Liu, Zhi Wang, Yan Zheng, Jianye Hao, Chenjia Bai, Junjie Ye, Zhen Wang, Haiyin Piao, and Yang Sun. Ovd-explorer: Optimism should not be the sole pursuit of exploration in noisy environments. In Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 13954-13962. AAAI Press, 2024.","title":"Ovd-explorer: Optimism should not be the sole pursuit of exploration in noisy environments","authors":"Jinyi Liu, Zhi Wang, Yan Zheng, Jianye Hao, Chenjia Bai, Junjie Ye, Zhen Wang, Haiyin Piao, Yang Sun","year":"2024","venue":"Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Louizos et al., 2017] Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. Advances in neural information processing systems, 30, 2017.","title":"Bayesian compression for deep learning","authors":"Christos Louizos, Karen Ullrich, Max Welling","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lu et al., 2021] Yao Lu, Karol Hausman, Yevgen Chebo- tar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, et al. Aw-opt: Learning robotic skills with imitation and reinforcement at scale. arXiv preprint arXiv:2111.05424, 2021.","title":"Aw-opt: Learning robotic skills with imitation and reinforcement at scale","authors":"Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Mao et al., 2022] Yihuan Mao, Chao Wang, Bin Wang, and Chongjie Zhang. Moore: Model-based offline-to-online reinforcement learning. arXiv preprint arXiv:2201.10070, 2022.","title":"Moore: Model-based offline-to-online reinforcement learning","authors":"Yihuan Mao, Chao Wang, Bin Wang, Chongjie Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mark et al., 2022] Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.","title":"Fine-tuning offline policies with optimistic action selection","authors":"Mark","year":"2022","venue":"Deep Reinforcement Learning Workshop NeurIPS 2022","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Nair et al., 2020] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online re- inforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.","title":"Awac: Accelerating online reinforcement learning with offline datasets","authors":"Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Nakamoto et al., 2023] Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arXiv:2303.05479, 2023.","title":"Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning","authors":"Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, Sergey Levine","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information pro- cessing systems, 29, 2016.","title":"Deep exploration via bootstrapped dqn","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Savinov et al., 2018] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Tim- othy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274, 2018.","title":"Episodic curiosity through reachability","authors":"Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Schrittwieser et al., 2020] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609, 2020.","title":"Mastering atari, go, chess and shogi by planning with a learned model","authors":"Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel","year":"2020","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Seno and Imai, 2022] Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. The Journal of Machine Learning Research, 23(1):14205- 14224, 2022.","title":"d3rlpy: An offline deep reinforcement learning library","authors":"Imai ; ] Takuma Seno, Michita Seno, Imai","year":"2022","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tarasov et al., 2022] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline rein- forcement learning library. In 3rd Offline RL Workshop: Offline RL as a \"Launchpad\", 2022.","title":"CORL: Research-oriented deep offline reinforcement learning library","authors":"Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, Sergey Kolesnikov","year":"2022","venue":"3rd Offline RL Workshop: Offline RL as a \"Launchpad","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In 2012 IEEE/RSJ international conference on intel- ligent robots and systems, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ international conference on intelligent robots and systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tsividis et al., 2021] Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J Gershman, and Joshua B Tenenbaum. Human- level reinforcement learning through theory-based mod- eling, exploration, and planning. arXiv preprint arXiv:2107.12544, 2021.","title":"Humanlevel reinforcement learning through theory-based modeling, exploration, and planning","authors":"Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Joshua B Samuel J Gershman, Tenenbaum","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Jun- young Chung, Michael Mathieu, Max Jaderberg, Woj- ciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2:20, 2019.","title":"Alphastar: Mastering the real-time strategy game starcraft ii","authors":"Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell","year":"2019","venue":"DeepMind blog","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wu et al., 2019] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.","title":"Behavior regularized offline reinforcement learning","authors":"Yifan Wu, George Tucker, Ofir Nachum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yu et al., 2020] Tianhe Yu, Garrett Thomas, Lantao Yu, Ste- fano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy op- timization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.","title":"Mopo: Model-based offline policy optimization","authors":"Yu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2023] Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online re- inforcement learning. arXiv preprint arXiv:2302.00935, 2023.","title":"Policy expansion for bridging offline-to-online reinforcement learning","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhao et al., 2022] Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behav- ior cloning regularization for stable offline-to-online re- inforcement learning. arXiv preprint arXiv:2210.13846, 2022.","title":"Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning","authors":"Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, Joni Pajarinen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}