{"type":"ScholarlyArticle","title":"Offline Reinforcement Learning with Behavioral Supervisor Tuning","authors":[{"name":"Padmanaba Srinivasan","slug":"padmanaba-srinivasan","given":"Padmanaba","family":"Srinivasan","affiliation":"Imperial College London"},{"name":"William Knottenbelt","slug":"william-knottenbelt","given":"William","family":"Knottenbelt","affiliation":"Imperial College London"}],"abstract":"Offline reinforcement learning (RL) algorithms are applied to learn performant, well-generalizing policies when provided with a static dataset of interactions. Many recent approaches to offline RL have seen substantial success, but with one key caveat: they demand substantial per-dataset hyperparameter tuning to achieve reported performance which requires policy rollouts in the environment to evaluate; this can rapidly become cumbersome. Furthermore, substantial tuning requirements can hamper the adoption of these algorithms in practical domains. In this paper, we present TD3 with Behavioral Supervisor Tuning (TD3-BST), an algorithm that trains an uncertainty model and uses it to guide the policy to select actions within the dataset support. TD3-BST can learn more effective policies from offline datasets compared to prior methods and achieves the best performance across challenging benchmarks without requiring per-dataset tuning.","doi":"10.24963/ijcai.2024/545","url":"https://ijcai.org/proceedings/2024/545","pdf":"https://ijcai.org/proceedings/2024/0545.pdf","markdown":"https://ijcai.org/proceedings/2024/545.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/545","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":545,"pages":{"first":4929,"last":4937},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Offline reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Amari, 2016] Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.","title":"Shun-ichi Amari. Information geometry and its applications","authors":"Amari","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An et al., 2021] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline rein- forcement learning with diversified Q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.","title":"Uncertainty-based offline reinforcement learning with diversified Q-ensemble","authors":"Seungyong Gaon An, Jang-Hyun Moon, Hyun Oh Kim, Song","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Basu and Prasad, 2020] Somnath Basu and Sachchidanand Prasad. A connection between cut locus, thom space and morse-bott functions. arXiv preprint arXiv:2011.02972, 2020.","title":"A connection between cut locus, thom space and morse-bott functions","authors":"Somnath Basu, Sachchidanand Prasad","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brandfonbrener et al., 2021] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. Advances in neural infor- mation processing systems, 34:4933-4946, 2021.","title":"Offline RL without off-policy evaluation","authors":"David Brandfonbrener, Will Whitney, Rajesh Ranganath, Joan Bruna","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dherin et al., 2023] Benoit Dherin, Huiyi Hu, Jie Ren, Michael W Dusenberry, and Balaji Lakshminarayanan. Morse neural networks for uncertainty quantification. arXiv preprint arXiv:2307.00667, 2023.","title":"Morse neural networks for uncertainty quantification","authors":"Huiyi Benoit Dherin, Jie Hu, Michael W Ren, Balaji Dusenberry, Lakshminarayanan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Emerson et al., 2023] Harry Emerson, Matthew Guy, and Ryan McConville. Offline reinforcement learning for safer blood glucose control in people with type 1 diabetes. Jour- nal of Biomedical Informatics, 142:104376, 2023.","title":"Offline reinforcement learning for safer blood glucose control in people with type 1 diabetes","authors":"Emerson","year":"2023","venue":"Journal of Biomedical Informatics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Florence et al., 2022] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168. PMLR, 2022.","title":"Implicit behavioral cloning","authors":"Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, Jonathan Tompson","year":"2022","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4RL: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fu et al., 2022] Yuwei Fu, Di Wu, and Benoit Boulet. A closer look at offline RL agents. Advances in Neural In- formation Processing Systems, 35:8591-8604, 2022.","title":"A closer look at offline RL agents","authors":"Yuwei Fu, Di Wu, Benoit Boulet","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fujimoto and Gu, 2021] Scott Fujimoto and Shixi- ang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.","title":"A minimalist approach to offline reinforcement learning","authors":"Gu Fujimoto, Scott Fujimoto, Shixiang Shane Gu","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on ma- chine learning, pages 1587-1596. PMLR, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on ma- chine learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ghasemipour et al., 2022] Kamyar Ghasemipour, Shixi- ang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:18267-18281, 2022.","title":"Why so pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters","authors":"Kamyar Ghasemipour, Shixiang Shane Gu, Ofir Nachum","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.","title":"Deep learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gulcehre et al., 2020] Caglar Gulcehre, Sergio Gómez Col- menarejo, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Addressing extrapolation error in deep offline reinforcement learning. 2020. [Hasselt, 2010] Hado Hasselt. Double Q-learning. Advances in neural information processing systems, 23, 2010.","title":"Addressing extrapolation error in deep offline reinforcement learning","authors":"Caglar Gulcehre, Sergio Gómez Colmenarejo, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, Nando De Freitas","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kondratyuk et al., 2020] Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong. When ensembling smaller models is more efficient than single large models. arXiv preprint arXiv:2005.00570, 2020.","title":"When ensembling smaller models is more efficient than single large models","authors":"Dan Kondratyuk, Mingxing Tan, Matthew Brown, Boqing Gong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kostrikov et al., 2021a] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforce- ment learning with Fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.","title":"Offline reinforcement learning with Fisher divergence critic regularization","authors":"Kostrikov","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kostrikov et al., 2021b] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with im- plicit Q-learning. arXiv preprint arXiv:2110.06169, 2021.","title":"Offline reinforcement learning with implicit Q-learning","authors":"Kostrikov","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kumar et al., 2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.","title":"Stabilizing off-policy Q-learning via bootstrapping error reduction","authors":"Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Infor- mation Processing Systems, 33:1179-1191, 2020.","title":"Conservative Q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lakshminarayanan et al., 2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensem- bles. Advances in neural information processing systems, 30, 2017.","title":"Simple and scalable predictive uncertainty estimation using deep ensembles","authors":"Alexander Balaji Lakshminarayanan, Charles Pritzel, Blundell","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lange et al., 2012] Sascha Lange, Thomas Gabel, and Mar- tin Riedmiller. Batch reinforcement learning, pages 45- 73. Springer, 2012.","title":"Batch reinforcement learning","authors":"Sascha Lange, Thomas Gabel, Martin Riedmiller","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lee et al., 2018] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detect- ing out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.","title":"A simple unified framework for detecting out-of-distribution samples and adversarial attacks","authors":"Lee","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lei et al., 2023] Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Uni- fying online and offline deep reinforcement learning with multi-step on-policy optimization. arXiv preprint arXiv:2311.03351, 2023.","title":"Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization","authors":"Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, Huazhe Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2022] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline rein- forcement learning. In The Eleventh International Confer- ence on Learning Representations, 2022.","title":"When data geometry meets deep function: Generalizing offline reinforcement learning","authors":"Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, Ya-Qin Zhang","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2023] Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. Offline reinforcement learning with closed-form policy improvement operators. In International Conference on Machine Learning, pages 20485-20528. PMLR, 2023.","title":"Offline reinforcement learning with closed-form policy improvement operators","authors":"Li","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mirowski et al., 2018] Piotr Mirowski, Matt Grimes, Ma- teusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, and Raia Hadsell. Learning to navigate in cities without a map. Advances in neural information processing systems, 31, 2018.","title":"Learning to navigate in cities without a map","authors":"Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Moskovitz et al., 2021] Ted Moskovitz, Jack Parker- Holder, Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:12849-12863, 2021.","title":"Tactical optimism and pessimism for deep reinforcement learning","authors":"Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, Michael Jordan","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Nair et al., 2020] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online re- inforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.","title":"AWAC: Accelerating online reinforcement learning with offline datasets","authors":"Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Nguyen et al., 2015] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High con- fidence predictions for unrecognizable images. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015.","title":"Deep neural networks are easily fooled: High confidence predictions for unrecognizable images","authors":"Anh Nguyen, Jason Yosinski, Jeff Clune","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Nikulin et al., 2023] Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. arXiv preprint arXiv:2301.13616, 2023.","title":"Anti-exploration by random network distillation","authors":"Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Sergey Kolesnikov","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Peng et al., 2019] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learn- ing. arXiv preprint arXiv:1910.00177, 2019.","title":"Advantage-weighted regression: Simple and scalable off-policy reinforcement learning","authors":"Xue Bin Peng, Aviral Kumar, Grace Zhang, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Rezaeifar et al., 2022] Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforce- ment learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8106-8114, 2022.","title":"Offline reinforcement learning as anti-exploration","authors":"Rezaeifar","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Rummery and Niranjan, 1994] Gavin A Rummery and Ma- hesan Niranjan. On-line Q-learning using connectionist systems, volume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994.","title":"On-line Q-learning using connectionist systems","authors":"A Gavin, Mahesan Rummery, Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Singh et al., 2022] Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, and Sergey Levine. Offline RL with realistic datasets: Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052, 2022.","title":"Offline RL with realistic datasets: Heteroskedasticity and support constraints","authors":"Singh","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tang and Wiens, 2021] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Prac- tical considerations for healthcare settings. In Machine Learning for Healthcare Conference, pages 2-35. PMLR, 2021.","title":"Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings","authors":"Wiens Tang","year":"2021","venue":"Machine Learning for Healthcare Conference","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Tarasov et al., 2022] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep of- fline reinforcement learning library. arXiv preprint arXiv:2210.07105, 2022.","title":"CORL: Research-oriented deep offline reinforcement learning library","authors":"Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, Sergey Kolesnikov","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tarasov et al., 2023] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2305.09836, 2023.","title":"Revisiting the minimalist approach to offline reinforcement learning","authors":"Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, Sergey Kolesnikov","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Jour- nal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-SNE","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Williams and Rasmussen, 2006] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for ma- chine learning, volume 2. MIT press Cambridge, MA, 2006.","title":"Gaussian processes for machine learning","authors":"K I Christopher, Carl Edward Williams, Rasmussen","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wu et al., 2019] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.","title":"Behavior regularized offline reinforcement learning","authors":"Yifan Wu, George Tucker, Ofir Nachum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Xu et al., 2023] Haoran Xu, Li Jiang, Jianxiong Li, Zhuo- ran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810, 2023.","title":"Offline RL with no OOD actions: In-sample learning via implicit value regularization","authors":"Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai, Kin Chan, Xianyuan Zhan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Yu et al., 2021] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.","title":"Reinforcement learning in healthcare: A survey","authors":"Chao Yu, Jiming Liu, Shamim Nemati, Guosheng Yin","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhang and Jiang, 2021] Siyuan Zhang and Nan Jiang. To- wards hyperparameter-free policy selection for offline re- inforcement learning. Advances in Neural Information Processing Systems, 34:12864-12875, 2021.","title":"Towards hyperparameter-free policy selection for offline reinforcement learning","authors":"Jiang ; ] Siyuan Zhang, Nan Zhang, Jiang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhang et al., 2023] Hongchang Zhang, Jianzhun Shao, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. DARL: distance-aware uncertainty estimation for offline rein- forcement learning. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 37, pages 11210- 11218, 2023.","title":"DARL: distance-aware uncertainty estimation for offline reinforcement learning","authors":"Hongchang Zhang, Jianzhun Shao, Shuncheng He, Yuhang Jiang, Xiangyang Ji","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhu et al., 2022] Deyao Zhu, Li Erran Li, and Mohamed El- hoseiny. Value memory graph: A graph-structured world model for offline reinforcement learning. arXiv preprint arXiv:2206.04384, 2022.","title":"Value memory graph: A graph-structured world model for offline reinforcement learning","authors":"Deyao Zhu, Li Erran Li, Mohamed Elhoseiny","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhu et al., 2023] Taiyu Zhu, Kezhi Li, and Pantelis Geor- giou. Offline deep reinforcement learning and off-policy evaluation for personalized basal insulin control in type 1 diabetes. IEEE Journal of Biomedical and Health Infor- matics, 2023.","title":"Offline deep reinforcement learning and off-policy evaluation for personalized basal insulin control in type 1 diabetes","authors":"Zhu","year":"2023","venue":"IEEE Journal of Biomedical and Health Informatics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}