{"type":"ScholarlyArticle","title":"vMFER: Von Mises-Fisher Experience Resampling Based on Uncertainty of Gradient Directions for Policy Improvement","authors":[{"name":"Yiwen Zhu","slug":"yiwen-zhu","given":"Yiwen","family":"Zhu","affiliation":"Zhejiang University; NetEase Fuxi AI Lab; Nanyang Technological University"},{"name":"Jinyi Liu","slug":"jinyi-liu","given":"Jinyi","family":"Liu","affiliation":"Tianjin University"},{"name":"Wenya Wei","slug":"wenya-wei","given":"Wenya","family":"Wei","affiliation":"Zhejiang University"},{"name":"Qianyi Fu","slug":"qianyi-fu","given":"Qianyi","family":"Fu","affiliation":"Zhejiang University"},{"name":"Yujing Hu","slug":"yujing-hu","given":"Yujing","family":"Hu","affiliation":"NetEase Fuxi AI Lab"},{"name":"Zhou Fang","slug":"zhou-fang","given":"Zhou","family":"Fang","affiliation":"Zhejiang University"},{"name":"Bo An","slug":"bo-an","given":"Bo","family":"An","affiliation":"Nanyang Technological University; Skywork AI"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"Tianjin University"},{"name":"Tangjie Lv","slug":"tangjie-lv","given":"Tangjie","family":"Lv","affiliation":"NetEase Fuxi AI Lab"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"NetEase Fuxi AI Lab"}],"abstract":"Reinforcement Learning (RL) is a widely employed technique in decision-making problems, encompassing two fundamental operations -- policy evaluation and policy improvement. Enhancing learning efficiency remains a key challenge in RL, with many efforts focused on using ensemble critics to boost policy evaluation efficiency. However, when using multiple critics, the actor in the policy improvement process can obtain different gradients. Previous studies have combined these gradients without considering their disagreements. Therefore, optimizing the policy improvement process is crucial to enhance learning efficiency. This study focuses on investigating the impact of gradient disagreements caused by ensemble critics on policy improvement. We introduce the concept of uncertainty of gradient directions as a means to measure the disagreement among gradients utilized in the policy improvement process. Through measuring the disagreement among gradients, we find that transitions with lower uncertainty of gradient directions are more reliable in the policy improvement process. Building on this analysis, we propose a method called von Mises-Fisher Experience Resampling (vMFER), which optimizes the policy improvement process by resampling transitions and assigning higher confidence to transitions with lower uncertainty of gradient directions. Our experiments demonstrate that vMFER significantly outperforms the benchmark and is particularly well-suited for ensemble structures in RL.","doi":"10.24963/ijcai.2024/633","url":"https://ijcai.org/proceedings/2024/633","pdf":"https://ijcai.org/proceedings/2024/0633.pdf","markdown":"https://ijcai.org/proceedings/2024/633.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/633","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":633,"pages":{"first":5725,"last":5733},"keywords":["Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Abbas et al., 2020] Zaheer Abbas, Samuel Sokota, Erin Talvitie, and Martha White. Selective dyna-style planning under limited model capacity. In International Conference on Machine Learning, pages 1-10, 2020.","title":"Selective dyna-style planning under limited model capacity","authors":"Zaheer Abbas, Samuel Sokota, Erin Talvitie, Martha White","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abeyruwan et al., 2023] Saminda Wishwajith Abeyruwan, Laura Graesser, David B D'Ambrosio, and et al. i- sim2real: Reinforcement learning of robotic policies in tight human-robot interaction loops. In Conference on Robot Learning, pages 212-224. PMLR, 2023.","title":"isim2real: Reinforcement learning of robotic policies in tight human-robot interaction loops","authors":"Abeyruwan","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Agarwal et al., 2020] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on of- fline reinforcement learning. In International Conference on Machine Learning, pages 104-114, 2020.","title":"An optimistic perspective on offline reinforcement learning","authors":"Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[An et al., 2021] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline rein- forcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems, 34:7436-7447, 2021.","title":"Uncertainty-based offline reinforcement learning with diversified q-ensemble","authors":"Seungyong Gaon An, Jang-Hyun Moon, Hyun Oh Kim, Song","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, and et al. Hindsight experience replay. Advances in Neural Information Pro- cessing Systems, 30, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Anschel et al., 2017] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabi- lization for deep reinforcement learning. In International Conference on Machine Learning, pages 176-185, 2017.","title":"Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning","authors":"Anschel","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Audibert et al., 2009] Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoret- ical Computer Science, 410(19):1876-1902, 2009.","title":"Exploration-exploitation tradeoff using variance estimates in multi-armed bandits","authors":"Jean-Yves Audibert, Rémi Munos, Csaba Szepesvári","year":"2009","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Banerjee et al., 2005] Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(9), 2005.","title":"Clustering on the unit hypersphere using von mises-fisher distributions","authors":"Arindam Banerjee, Joydeep Inderjit S Dhillon, Suvrit Ghosh, Greg Sra, Ridgeway","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bingham, 1974] Christopher Bingham. An antipodally symmetric distribution on the sphere. The Annals of Statis- tics, pages 1201-1225, 1974.","title":"An antipodally symmetric distribution on the sphere","authors":"Christopher Bingham","year":"1974","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bowman, 2012] Frank Bowman. Introduction to Bessel functions. Courier Corporation, 2012.","title":"Introduction to Bessel functions","authors":"Frank Bowman","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Buckman et al., 2018] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in Neural Informa- tion Processing Systems, 31, 2018.","title":"Sample-efficient reinforcement learning with stochastic ensemble value expansion","authors":"Buckman","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2017] Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q- ensembles. arXiv preprint arXiv:1706.01502, 2017.","title":"Ucb exploration via qensembles","authors":"Szymon Richard Y Chen, Pieter Sidor, John Abbeel, Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Chen et al., 2021] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q- learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.","title":"Randomized ensembled double qlearning: Learning fast without a model","authors":"Xinyue Chen, Che Wang, Zijian Zhou, Keith Ross","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ciosek et al., 2019] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with op- timistic actor critic. Advances in Neural Information Pro- cessing Systems, 32, 2019.","title":"Better exploration with optimistic actor critic","authors":"Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Clements et al., 2019] William R Clements, Bastien Van Delft, Benoît-Marie Robaglia, and et al. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.09638, 2019.","title":"Estimating risk and uncertainty in deep reinforcement learning","authors":"Bastien William R Clements, Benoît-Marie Van Delft, Robaglia","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Dearden et al., 1998] Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. AAAI, 1998:761-768, 1998.","title":"Bayesian q-learning","authors":"Richard Dearden, Nir Friedman, Stuart Russell","year":"1998","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Fisher, 1953] Ronald Aylmer Fisher. Dispersion on a sphere. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 217(1130):295- 305, 1953.","title":"Dispersion on a sphere","authors":"Aylmer Ronald, Fisher","year":"1130","venue":"Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Haarnoja et al., 2018a] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Haarnoja","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Haarnoja et al., 2018b] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Haarnoja","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Hasselt, 2010] Hado Hasselt. Double q-learning. Advances in Neural Information Processing Systems, 23, 2010.","title":"Hado Hasselt. Double q-learning","authors":"Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kalweit and Boedecker, 2017] Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continu- ous deep reinforcement learning. In Conference on Robot Learning, pages 195-206, 2017.","title":"Uncertainty-driven imagination for continuous deep reinforcement learning","authors":"Boedecker Kalweit, Gabriel Kalweit, Joschka Boedecker","year":"2017","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kent, 1982] John T Kent. The fisher-bingham distribution on the sphere. Journal of the Royal Statistical Society: Series B (Methodological), 44(1):71-80, 1982.","title":"The fisher-bingham distribution on the sphere","authors":"T John, Kent","year":"1982","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kidambi et al., 2020] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 33:21810-21823, 2020.","title":"Morel: Model-based offline reinforcement learning","authors":"Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lan et al., 2020] Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Con- trolling the estimation bias of q-learning. arXiv preprint arXiv:2002.06487, 2020.","title":"Maxmin q-learning: Controlling the estimation bias of q-learning","authors":"Qingfeng Lan, Yangchen Pan, Alona Fyshe, Martha White","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lee et al., 2021] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learn- ing, pages 6131-6141. PMLR, 2021.","title":"Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning","authors":"Lee","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lee et al., 2022] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online re- inforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.","title":"Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble","authors":"Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, Jinwoo Shin","year":"2022","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Liu et al., 2024] Jinyi Liu, Zhi Wang, Yan Zheng, Jianye Hao, Chenjia Bai, Junjie Ye, Zhen Wang, Haiyin Piao, and Yang Sun. Ovd-explorer: Optimism should not be the sole pursuit of exploration in noisy environments. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 38(12):13954-13962, Mar. 2024.","title":"Ovd-explorer: Optimism should not be the sole pursuit of exploration in noisy environments","authors":"Jinyi Liu, Zhi Wang, Yan Zheng, Jianye Hao, Chenjia Bai, Junjie Ye, Zhen Wang, Haiyin Piao, Yang Sun","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Lockwood and Si, 2022] Owen Lockwood and Mei Si. A review of uncertainty for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence and Interactive Digital Entertainment, volume 18, pages 155-162, 2022.","title":"A review of uncertainty for deep reinforcement learning","authors":"Lockwood, Si ;, Owen Lockwood, Mei Si","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Mardia et al., 2000] Kanti V Mardia, Peter E Jupp, and KV Mardia. Directional Statistics, volume 2. Wiley On- line Library, 2000.","title":"Directional Statistics","authors":"Peter E Kanti V Mardia, Jupp, Mardia","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in Neural Information Pro- cessing Systems, 29, 2016.","title":"Deep exploration via bootstrapped dqn","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Osband et al., 2018] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforce- ment learning. Advances in Neural Information Process- ing Systems, 31, 2018.","title":"Randomized prior functions for deep reinforcement learning","authors":"Ian Osband, John Aslanides, Albin Cassirer","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Plappert et al., 2018] Matthias Plappert, Marcin Andrycho- wicz, Alex Ray, and et al. Multi-goal reinforcement learn- ing: Challenging robotics environments and request for re- search. arXiv preprint arXiv:1802.09464, 2018.","title":"Multi-goal reinforcement learning: Challenging robotics environments and request for research","authors":"Matthias Plappert, Marcin Andrychowicz, Alex Ray","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Radosavovic et al., 2023] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre- training. In Conference on Robot Learning, pages 416- 426, 2023.","title":"Real-world robot learning with masked visual pretraining","authors":"Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, Trevor Darrell","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Shen and How, 2021] Macheng Shen and Jonathan P How. Robust opponent modeling via adversarial ensemble rein- forcement learning. In Proceedings of the International Conference on Automated Planning and Scheduling, vol- ume 31, pages 578-587, 2021.","title":"Robust opponent modeling via adversarial ensemble reinforcement learning","authors":"How Shen, Macheng Shen, Jonathan P How","year":"2021","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Song et al., 2023] Yanjie Song, PN Suganthan, Witold Pedrycz, Junwei Ou, Yongming He, and Yingwu Chen. Ensemble reinforcement learning: A survey. arXiv preprint arXiv:2303.02618, 2023.","title":"Ensemble reinforcement learning: A survey","authors":"Song","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Sra, 2012] Suvrit Sra. A short note on parameter approxi- mation for von mises-fisher distributions: and a fast imple- mentation of I s (x). Computational Statistics, 27:177-190, 2012.","title":"Suvrit Sra. A short note on parameter approximation for von mises-fisher distributions: and a fast implementation of I s (x)","authors":"Sra","year":"2012","venue":"Computational Statistics","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Thrun and Schwartz, 1993] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for re- inforcement learning. In Proceedings of the Fourth Con- nectionist Models Summer School, volume 255, page 263.","title":"Issues in using function approximation for reinforcement learning","authors":"Schwartz Thrun, Sebastian Thrun, Anton Schwartz","year":"1993","venue":"Proceedings of the Fourth Connectionist Models Summer School","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wang and Zou, 2021] Yue Wang and Shaofeng Zou. On- line robust reinforcement learning with model uncer- tainty. Advances in Neural Information Processing Sys- tems, 34:7193-7206, 2021.","title":"Online robust reinforcement learning with model uncertainty","authors":"Yue Wang, Shaofeng Zou","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Watson, 1982] Geoffrey S Watson. Distributions on the circle and sphere. Journal of Applied Probability, 19(A):265-280, 1982.","title":"Distributions on the circle and sphere","authors":"Geoffrey S Watson","year":"1982","venue":"Journal of Applied Probability","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wu et al., 2022] Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, and Hankz Hankui Zhuo. Plan to predict: Learn- ing an uncertainty-foreseeing model for model-based rein- forcement learning. Advances in Neural Information Pro- cessing Systems, 35:15849-15861, 2022.","title":"Plan to predict: Learning an uncertainty-foreseeing model for model-based reinforcement learning","authors":"Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, Hankz Hankui, Zhuo","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Wu et al., 2023] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, pages 2226-2240, 2023.","title":"Daydreamer: World models for physical robot learning","authors":"Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, Ken Goldberg","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Yang et al., 2021] Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: a comprehensive survey. arXiv preprint arXiv:2109.06668, 2021.","title":"Exploration in deep reinforcement learning: a comprehensive survey","authors":"Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Yu et al., 2020] Tianhe Yu, Garrett Thomas, Lantao Yu, Ste- fano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy op- timization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.","title":"Mopo: Model-based offline policy optimization","authors":"Yu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhang et al., 2020] Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Genc, Sunil Mallya, and Tamer Basar. Ro- bust multi-agent reinforcement learning with model uncer- tainty. Advances in Neural Information Processing Sys- tems, 33:10571-10583, 2020.","title":"Robust multi-agent reinforcement learning with model uncertainty","authors":"Zhang","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhao et al., 2023] Kai Zhao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. Ensemble-based offline-to-online re- inforcement learning: From pessimistic learning to opti- mistic exploration. CoRR, abs/2306.06871, 2023.","title":"Ensemble-based offline-to-online reinforcement learning: From pessimistic learning to optimistic exploration","authors":"Kai Zhao, Yi Ma, Jinyi Liu, Yan Zheng, Zhaopeng Meng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}