{"type":"ScholarlyArticle","title":"MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator","authors":[{"name":"Xiao-Yin Liu","slug":"xiao-yin-liu","given":"Xiao-Yin","family":"Liu","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Xiao-Hu Zhou","slug":"xiao-hu-zhou","given":"Xiao-Hu","family":"Zhou","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Guotao Li","slug":"guotao-li","given":"Guotao","family":"Li","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Hao Li","slug":"hao-li","given":"Hao","family":"Li","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Mei-Jiang Gui","slug":"mei-jiang-gui","given":"Mei-Jiang","family":"Gui","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Tian-Yu Xiang","slug":"tian-yu-xiang","given":"Tian-Yu","family":"Xiang","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"De-Xing Huang","slug":"de-xing-huang","given":"De-Xing","family":"Huang","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"},{"name":"Zeng-Guang Hou","slug":"zeng-guang-hou","given":"Zeng-Guang","family":"Hou","affiliation":"State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation , Chinese Academy of Sciences , Beijing 100190 , China; The School of Artificial Intelligence University of Chinese Academy of Sciences , Beijing 100049 , China"}],"abstract":"Offline reinforcement learning (RL) faces a significant challenge of distribution shift. Model-free offline RL penalizes the Q value for out-of-distribution (OOD) data or constrains the policy closed to the behavior policy to tackle this problem, but this inhibits the exploration of the OOD region. Model-based offline RL, which uses the trained environment model to generate more OOD data and performs conservative policy optimization within that model, has become an effective method for this problem. However, the current model-based algorithms rarely consider agent robustness when incorporating conservatism into policy. Therefore, the new model-based offline algorithm with a conservative Bellman operator (MICRO) is proposed. This method trades off performance and robustness via introducing the robust Bellman operator into the algorithm. Compared with previous model-based algorithms with robust adversarial models, MICRO can significantly reduce the computation cost by only choosing the minimal Q value in the state uncertainty set. Extensive experiments demonstrate that MICRO outperforms prior RL algorithms in offline RL benchmark and is considerably robust to adversarial perturbations.","doi":"10.24963/ijcai.2024/507","url":"https://ijcai.org/proceedings/2024/507","pdf":"https://ijcai.org/proceedings/2024/0507.pdf","markdown":"https://ijcai.org/proceedings/2024/507.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/507","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":507,"pages":{"first":4587,"last":4595},"keywords":["Machine Learning: ML: Model-based and model learning reinforcement learning","Machine Learning: ML: Robustness","Machine Learning: ML: Offline reinforcement learning","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Abdullah et al., 2019] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust re- inforcement learning. arXiv preprint arXiv:1907.13196, 2019.","title":"Wasserstein robust reinforcement learning","authors":"Mohammed Amin, Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, Jun Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An et al., 2021] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline rein- forcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems, 34:7436-7447, 2021.","title":"Uncertainty-based offline reinforcement learning with diversified q-ensemble","authors":"Seungyong Gaon An, Jang-Hyun Moon, Hyun Oh Kim, Song","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Badrinath and Kalathil, 2021] Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learn- ing, pages 511-520. PMLR, 2021.","title":"Robust reinforcement learning using least squares policy iteration with provable performance guarantees","authors":"Kalathil ; Badrinath, Panaganti Kishan, Dileep Badrinath, Kalathil","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2021] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2021.","title":"Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning","authors":"Bai","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bhardwaj et al., 2023] Mohak Bhardwaj, Tengyang Xie, By- ron Boots, Nan Jiang, and Ching-An Cheng. Adversarial model for offline reinforcement learning. Advances in Neu- ral Information Processing Systems, 2023.","title":"Adversarial model for offline reinforcement learning","authors":"Bhardwaj","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blanchet and Murthy, 2019] Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565- 600, 2019.","title":"Quantifying distributional model risk via optimal transport","authors":"Murthy Blanchet, Jose Blanchet, Karthyek Murthy","year":"2019","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Jiang, 2019] Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforce- ment learning. In International Conference on Machine Learning, pages 1042-1051. PMLR, 2019.","title":"Information-theoretic considerations in batch reinforcement learning","authors":"Jinglin Chen, Nan Jiang","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cheng et al., 2022] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Con- ference on Machine Learning, pages 3852-3878. PMLR, 2022.","title":"Adversarially trained actor critic for offline reinforcement learning","authors":"Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning with- out exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Haarnoja","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hou et al., 2020] Linfang Hou, Liang Pang, Xin Hong, Yanyan Lan, Zhiming Ma, and Dawei Yin. Robust re- inforcement learning with wasserstein constraint. arXiv preprint arXiv:2006.00945, 2020.","title":"Robust reinforcement learning with wasserstein constraint","authors":"Hou","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Iyengar, 2005] Garud N Iyengar. Robust dynamic program- ming. Mathematics of Operations Research, 30(2):257- 280, 2005.","title":"Robust dynamic programming","authors":"N Garud, Iyengar","year":"2005","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Janner et al., 2019] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural In- formation Processing Systems, 32, 2019.","title":"When to trust your model: Model-based policy optimization","authors":"Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kamalaruban et al., 2020] Parameswaran Kamalaruban, Yu- Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adver- sarial training with langevin dynamics. Advances in Neural Information Processing Systems, 33:8127-8138, 2020.","title":"Robust reinforcement learning via adversarial training with langevin dynamics","authors":"Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, Volkan Cevher","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim and Oh, 2023] Byeongchan Kim and Min-hwan Oh. Model-based offline reinforcement learning with count- based conservatism. In International Conference on Ma- chine Learning, 2023.","title":"Byeongchan Kim and Min-hwan Oh. Model-based offline reinforcement learning with countbased conservatism","authors":"Oh Kim","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kostrikov et al., 2021] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learn- ing with fisher divergence critic regularization. In Interna- tional Conference on Machine Learning, pages 5774-5783.","title":"Offline reinforcement learning with fisher divergence critic regularization","authors":"Kostrikov","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kuang et al., 2022] Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Learning robust pol- icy against disturbance in transition dynamics via state- conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7247-7254, 2022.","title":"Learning robust policy against disturbance in transition dynamics via stateconservative policy optimization","authors":"Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, Houqiang Li","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kumar et al., 2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.","title":"Stabilizing off-policy q-learning via bootstrapping error reduction","authors":"Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Infor- mation Processing Systems, 33:1179-1191, 2020.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.","title":"Offline reinforcement learning: Tutorial, review, and perspectives on open problems","authors":"Sergey Levine, Aviral Kumar, George Tucker, Justin Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2023] Hao Li, Xiao-Hu Zhou, Xiao-Liang Xie, Shi- Qi Liu, Zhen-Qiu Feng, Xiao-Yin Liu, Mei-Jiang Gui, Tian- Yu Xiang, De-Xing Huang, Bo-Xian Yao, et al. Crop: Conservative reward for model-based offline policy opti- mization. arXiv preprint arXiv:2310.17245, 2023.","title":"Crop: Conservative reward for model-based offline policy optimization","authors":"Hao Li, Xiao-Hu Zhou, Xiao-Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Xiao-Yin Liu, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Bo-Xian Yao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2023] Xiao-Yin Liu, Xiao-Hu Zhou, Xiao-Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, and Zeng-Guang Hou. Do- main: Mildly conservative model-based offline reinforce- ment learning. arXiv preprint arXiv:2309.08925, 2023.","title":"Domain: Mildly conservative model-based offline reinforcement learning","authors":"Xiao-Yin Liu, Xiao-Hu Zhou, Xiao-Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Zeng-Guang Hou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nilim and El Ghaoui, 2005] Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.","title":"Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices","authors":"El Nilim, Ghaoui","year":"2005","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Niu et al., 2022] Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-36612, 2022.","title":"When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning","authors":"Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, H U Jianming, Xianyuan Zhan","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Panaganti et al., 2022] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforce- ment learning using offline data. Advances in Neural Infor- mation Processing Systems, 35:32211-32224, 2022.","title":"Robust reinforcement learning using offline data","authors":"Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, Mohammad Ghavamzadeh","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pattanaik et al., 2018] Anay Pattanaik, Zhenyi Tang, Shui- jing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial at- tacks. In Proceedings of the 17th International Confer- ence on Autonomous Agents and MultiAgent Systems, pages 2040-2042, 2018.","title":"Robust deep reinforcement learning with adversarial attacks","authors":"Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, Girish Chowdhary","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pinto et al., 2017] Lerrel Pinto, James Davidson, Rahul Suk- thankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pages 2817-2826. PMLR, 2017.","title":"Robust adversarial reinforcement learning","authors":"Pinto","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rigter et al., 2022] Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 35:16082-16097, 2022.","title":"Rambo-rl: Robust adversarial model-based offline reinforcement learning","authors":"Rigter","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Singh et al., 2022] Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelli- gence Review, pages 1-46, 2022.","title":"Reinforcement learning in robotic applications: a comprehensive survey","authors":"Bharat Singh, Rajesh Kumar, Vinay Pratap Singh","year":"2022","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sun et al., 2023] Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learn- ing. In International Conference on Machine Learning, 2023.","title":"Model-bellman inconsistency for model-based offline reinforcement learning","authors":"Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, Yang Yu","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tessler et al., 2019] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In International Conference on Machine Learning, pages 6215-6224. PMLR, 2019.","title":"Action robust reinforcement learning and applications in continuous control","authors":"Chen Tessler, Yonathan Efroni, Shie Mannor","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Uehara and Sun, 2021] Masatoshi Uehara and Wen Sun. Pes- simistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226, 2021.","title":"Pessimistic model-based offline reinforcement learning under partial coverage","authors":"Masatoshi Uehara, Wen Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang and Zou, 2021] Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Ad- vances in Neural Information Processing Systems, 34:7193- 7206, 2021.","title":"Online robust reinforcement learning with model uncertainty","authors":"Yue Wang, Shaofeng Zou","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2021] Ruosong Wang, Dean Foster, and Sham M Kakade. What are the statistical limits of offline rl with linear function approximation? In International Conference on Learning Representations, 2021.","title":"What are the statistical limits of offline rl with linear function approximation?","authors":"Wang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xie et al., 2021] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:6683-6694, 2021.","title":"Bellman-consistent pessimism for offline reinforcement learning","authors":"Xie","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yang et al., 2022] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smooth- ing. Advances in Neural Information Processing Systems, 35:23851-23866, 2022.","title":"Rorl: Robust offline reinforcement learning via conservative smoothing","authors":"Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, Lei Han","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yu et al., 2020] Tianhe Yu, Garrett Thomas, Lantao Yu, Ste- fano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy op- timization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.","title":"Mopo: Model-based offline policy optimization","authors":"Yu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yu et al., 2021a] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.","title":"Reinforcement learning in healthcare: A survey","authors":"Yu","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yu et al., 2021b] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimiza- tion. Advances in Neural Information Processing Systems, 34:28954-28967, 2021.","title":"Combo: Conservative offline model-based policy optimization","authors":"Yu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhang et al., 2020] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adver- sarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024-21037, 2020.","title":"Robust deep reinforcement learning against adversarial perturbations on state observations","authors":"Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, Cho-Jui Hsieh","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}