{"type":"ScholarlyArticle","title":"Adaptive Reward Shifting Based on Behavior Proximity for Offline Reinforcement Learning","authors":[{"name":"Zhe Zhang","slug":"zhe-zhang","given":"Zhe","family":"Zhang","affiliation":"College of Computer Science and Technology , Nanjing University of Aeronautics and Astronautics; MIIT Key Laboratory of Pattern Analysis and Machine Intelligence"},{"name":"Xiaoyang Tan","slug":"xiaoyang-tan","given":"Xiaoyang","family":"Tan","affiliation":"College of Computer Science and Technology , Nanjing University of Aeronautics and Astronautics; MIIT Key Laboratory of Pattern Analysis and Machine Intelligence"}],"abstract":"One of the major challenges of the current offline reinforcement learning research is to deal with the distribution shift problem due to the change in state-action visitations for the new policy. To address this issue, we present a novel reward shifting-based method. Specifically, to regularize the behavior of the new policy at each state, we modify the reward to be received by the new policy by shifting it adaptively according to its proximity to the behavior policy, and apply the reward shifting along opposite directions for in-distribution actions and the ones not. In this way we are able to guide the learning procedure of the new policy itself by influencing the consequence of its actions explicitly, helping it to achieve a better balance between behavior constraints and policy improvement. Empirical results on the popular D4RL benchmarks show that the proposed method obtains competitive performance compared to the state-of-art baselines.","doi":"10.24963/ijcai.2023/514","url":"https://ijcai.org/proceedings/2023/514","pdf":"https://ijcai.org/proceedings/2023/0514.pdf","markdown":"https://ijcai.org/proceedings/2023/514.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/514","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":514,"pages":{"first":4620,"last":4628},"keywords":["Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Cabi et al., 2020] Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott E. Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Bud- den, Mel Vecerík, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Robotics: Science and Systems XVI, Virtual Event / Corvalis, Oregon, USA, July 12-16, 2020, 2020.","title":"Scaling data-driven robotics with reward sketching and batch reinforcement learning","authors":"Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott E Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerík, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando De Freitas, Ziyu Wang","year":"2020","venue":"Robotics: Science and Systems XVI, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fujimoto and Gu, 2021] Scott Fujimoto and Shixi- ang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.","title":"A minimalist approach to offline reinforcement learning","authors":"Gu Fujimoto, Scott Fujimoto, Shixiang Shane Gu","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on ma- chine learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Grigorescu et al., 2020] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386, 2020.","title":"A survey of deep learning techniques for autonomous driving","authors":"Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, Gigel Macesanu","year":"2020","venue":"Journal of Field Robotics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Konyushkova et al., 2020] Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas. Semi-supervised re- ward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06899, 2020.","title":"Semi-supervised reward learning for offline reinforcement learning","authors":"Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, Nando De Freitas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kostrikov et al., 2021] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforce- ment learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.","title":"Offline reinforcement learning with fisher divergence critic regularization","authors":"Kostrikov","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kostrikov et al., 2022] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with im- plicit q-learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.","title":"Offline reinforcement learning with implicit q-learning","authors":"Kostrikov","year":"2022","venue":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kumar et al., 2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.","title":"Stabilizing off-policy q-learning via bootstrapping error reduction","authors":"Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Infor- mation Processing Systems, 33:1179-1191, 2020.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Laud, 2004] Adam Daniel Laud. Theory and application of reward shaping in reinforcement learning. University of Illinois at Urbana-Champaign, 2004.","title":"Theory and application of reward shaping in reinforcement learning","authors":"Adam Laud, Laud Daniel","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.","title":"Offline reinforcement learning: Tutorial, review, and perspectives on open problems","authors":"Sergey Levine, Aviral Kumar, George Tucker, Justin Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mezghani et al., 2022] Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric, and Karteek Alahari. Learning goal-conditioned poli- cies offline with self-supervised reward shaping. In CoRL-Conference on Robot Learning, 2022.","title":"Learning goal-conditioned policies offline with self-supervised reward shaping","authors":"Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric, Karteek Alahari","year":"2022","venue":"CoRL-Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, vol- ume 99, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2019] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learn- ing. arXiv preprint arXiv:1910.00177, 2019.","title":"Advantage-weighted regression: Simple and scalable off-policy reinforcement learning","authors":"Xue Bin Peng, Aviral Kumar, Grace Zhang, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rezaeifar et al., 2022] Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforce- ment learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8106-8114, 2022.","title":"Offline reinforcement learning as anti-exploration","authors":"Rezaeifar","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Siegel et al., 2020] Noah Y Siegel, Jost Tobias Springen- berg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behav- ioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.","title":"Keep doing what worked: Behavioral modelling priors for offline reinforcement learning","authors":"Y Noah, Jost Siegel, Felix Tobias Springenberg, Abbas Berkenkamp, Michael Abdolmaleki, Thomas Neunert, Roland Lampe, Nicolas Hafner, Martin Heess, Riedmiller","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sohn et al., 2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural infor- mation processing systems, 28, 2015.","title":"Learning structured output representation using deep conditional generative models","authors":"Sohn","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2022] Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Optimistic curiosity ex- ploration and conservative exploitation with linear reward shaping. In Advances in Neural Information Processing Systems, 2022.","title":"Optimistic curiosity exploration and conservative exploitation with linear reward shaping","authors":"Lei Hao Sun, Rui Han, Xiaoteng Yang, Jian Ma, Bolei Guo, Zhou","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2019] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.","title":"Behavior regularized offline reinforcement learning","authors":"Yifan Wu, George Tucker, Ofir Nachum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2021] Yue Wu, Shuangfei Zhai, Nitish Srivas- tava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhut- dinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, 18- 24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11319-11328. PMLR, 2021.","title":"Uncertainty weighted actor-critic for offline reinforcement learning","authors":"Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning, ICML 2021","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2020] Tianhe Yu, Garrett Thomas, Lantao Yu, Ste- fano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy op- timization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.","title":"Mopo: Model-based offline policy optimization","authors":"Yu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2021a] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.","title":"Reinforcement learning in healthcare: A survey","authors":"Yu","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2021b] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy opti- mization. Advances in neural information processing sys- tems, 34:28954-28967, 2021.","title":"Combo: Conservative offline model-based policy optimization","authors":"Yu","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}