{"type":"ScholarlyArticle","title":"Adaptive Estimation Q-learning with Uncertainty and Familiarity","authors":[{"name":"Xiaoyu Gong","slug":"xiaoyu-gong","given":"Xiaoyu","family":"Gong","affiliation":"Key Laboratory of Symbolic Computation and Knowledge Engineering ( Jilin University) , Ministry of Education , China; College of Computer Science and Technology , Jilin University , China; College of Software , Jilin University , China"},{"name":"Shuai Lü","slug":"shuai-lu","given":"Shuai","family":"Lu","affiliation":""},{"name":"Jiayu Yu","slug":"jiayu-yu","given":"Jiayu","family":"Yu","affiliation":"Key Laboratory of Symbolic Computation and Knowledge Engineering ( Jilin University) , Ministry of Education , China; College of Computer Science and Technology , Jilin University , China"},{"name":"Sheng Zhu","slug":"sheng-zhu","given":"Sheng","family":"Zhu","affiliation":"Key Laboratory of Symbolic Computation and Knowledge Engineering ( Jilin University) , Ministry of Education , China; College of Software , Jilin University , China"},{"name":"Zongze Li","slug":"zongze-li","given":"Zongze","family":"Li","affiliation":"Key Laboratory of Symbolic Computation and Knowledge Engineering ( Jilin University) , Ministry of Education , China; College of Computer Science and Technology , Jilin University , China"}],"abstract":"One of the key problems in model-free deep reinforcement learning is how to obtain more accurate value estimations. Current most widely-used off-policy algorithms suffer from over- or underestimation bias which may lead to unstable policy. In this paper, we propose a novel method, Adaptive Estimation Q-learning (AEQ), which uses uncertainty and familiarity to control the value estimation naturally and can adaptively change for specific state-action pair. We theoretically prove the property of our familiarity term which can even keep the expected estimation bias approximate to 0, and experimentally demonstrate our dynamic estimation can improve the performance and prevent the bias continuously increasing. We evaluate AEQ on several continuous control tasks, outperforming state-of-the-art performance. Moreover, AEQ is simple to implement and can be applied in any off-policy actor-critic algorithm.","doi":"10.24963/ijcai.2023/417","url":"https://ijcai.org/proceedings/2023/417","pdf":"https://ijcai.org/proceedings/2023/0417.pdf","markdown":"https://ijcai.org/proceedings/2023/417.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/417","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":417,"pages":{"first":3750,"last":3758},"keywords":["Machine Learning: ML: Ensemble methods","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2020] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), pages 104-114, 2020.","title":"An optimistic perspective on offline reinforcement learning","authors":"Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning (ICML 2020)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An et al., 2021] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline rein- forcement learning with diversified Q-ensemble. In Pro- ceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Sydney, Australia, 2021.","title":"Uncertainty-based offline reinforcement learning with diversified Q-ensemble","authors":"Seungyong Gaon An, Jang-Hyun Moon, Hyun Oh Kim, Song","year":"2021","venue":"Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anschel et al., 2017] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabi- lization for deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICML 2017), pages 176-185, Sydney, NSW, Australia, 2017.","title":"Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning","authors":"Anschel","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML 2017)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Proceedings of the 30st Conference on Neu- ral Information Processing Systems (NIPS 2016), pages 1471-1479, Barcelona, Spain, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Proceedings of the 30st Conference on Neural Information Processing Systems (NIPS 2016)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Burda et al., 2019] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random net- work distillation. In Proceedings of the 7th International Conference on Learning Representations (ICLR 2019), New Orleans, LA, USA, 2019.","title":"Exploration by random network distillation","authors":"Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov","year":"2019","venue":"Proceedings of the 7th International Conference on Learning Representations (ICLR 2019)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017] Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB exploration via Q- ensembles. arXiv preprint arXiv:1706.01502, 2017.","title":"UCB exploration via Qensembles","authors":"Szymon Richard Y Chen, Pieter Sidor, John Abbeel, Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2021] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double Q- learning: Learning fast without a model. In Proceedings of the 9th International Conference on Learning Representa- tions (ICLR 2021), 2021.","title":"Randomized ensembled double Qlearning: Learning fast without a model","authors":"Xinyue Chen, Che Wang, Zijian Zhou, Keith Ross","year":"2021","venue":"Proceedings of the 9th International Conference on Learning Representations (ICLR 2021)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cideron et al., 2020] Geoffrey Cideron, Thomas Pierrot, Nicolas Perrin, Karim Beguir, and Olivier Sigaud. QD-RL: Efficient mixing of quality and diversity in reinforcement learning. arXiv preprint arXiv:2006.08505, pages 28-73, 2020.","title":"QD-RL: Efficient mixing of quality and diversity in reinforcement learning","authors":"Cideron","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ciosek et al., 2019] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with op- timistic actor-critic. In Proceedings of the 32nd Confer- ence on Neural Information Processing Systems (NeurIPS 2019), pages 1785-1796, Vancouver, BC, Canada, 2019.","title":"Better exploration with optimistic actor-critic","authors":"Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann","year":"2019","venue":"Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2019)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Conti et al., 2018] Edoardo Conti, Vashisht Madhavan, Fe- lipe Petroski Such, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Improving exploration in evolution strate- gies for deep reinforcement learning via a population of novelty-seeking agents. In Proceedings of the 32nd In- ternational Conference on Neural Information Process- ing Systems (NeurIPS 2018), pages 5032-5043, Montréal, Canada, 2018.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents","authors":"Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2018","venue":"Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS 2018)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Duan et al., 2021] Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems, 2021.","title":"Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors","authors":"Jingliang Duan, Yang Guan, Eben Shengbo, Yangang Li, Qi Ren, Bo Sun, Cheng","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th Inter- national Conference on Machine Learning (ICML 2018), pages 1582-1591, Stockholmsmässan, Stockholm, Swe- den, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML 2018)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th Inter- national Conference on Machine Learning (ICML 2018), pages 1856-1865, Stockholmsmässan, Stockholm, Swe- den, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML 2018)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hasselt, 2010] Hado Hasselt. Double Q-learning. In Pro- ceedings of the 24th Conference on Neural Information Processing Systems (NIPS 2010), pages 2613-2621, Van- couver, British Columbia, Canada, 2010.","title":"Hado Hasselt. Double Q-learning","authors":"Hasselt","year":"2010","venue":"Proceedings of the 24th Conference on Neural Information Processing Systems (NIPS 2010)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He and Hou, 2020] Qiang He and Xinwen Hou. WD3: Taming the estimation bias in deep reinforcement learn- ing. In 32nd IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2020), pages 391-398, Bal- timore, MD, USA, 2020.","title":"WD3: Taming the estimation bias in deep reinforcement learning","authors":"Hou ; ] Qiang He, Xinwen He, Hou","year":"2020","venue":"32nd IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2020)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kuznetsov et al., 2020] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), pages 5556-5566, 2020.","title":"Controlling overestimation bias with truncated mixture of continuous distributional quantile critics","authors":"Kuznetsov","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning (ICML 2020)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lan et al., 2020] Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning: Control- ling the estimation bias of Q-learning. In Proceedings of the 8th International Conference on Learning Representa- tions (ICLR 2020), Addis Ababa, Ethiopia, 2020.","title":"Maxmin Q-learning: Controlling the estimation bias of Q-learning","authors":"Qingfeng Lan, Yangchen Pan, Alona Fyshe, Martha White","year":"2020","venue":"Proceedings of the 8th International Conference on Learning Representations (ICLR 2020)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2021] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. In Proceedings of the 38th International Con- ference on Machine Learning (ICML 2021), pages 6131- 6141, 2021.","title":"SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning","authors":"Lee","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning (ICML 2021)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li and Hou, 2019] Zhunan Li and Xinwen Hou. Mixing up- date Q-value for deep reinforcement learning. In Proceed- ings of the 2019 International Joint Conference on Neural Networks (IJCNN 2019), pages 1-6, Budapest, Hungary, 2019.","title":"Mixing update Q-value for deep reinforcement learning","authors":"Hou ; ] Zhunan Li, Xinwen Li, Hou","year":"2019","venue":"Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN 2019)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2021] Sicen Li, Gang Wang, Qinyun Tang, and Liquan Wang. Balancing value underestimation and overestimation with realistic actor-critic. arXiv preprint arXiv:2110.09712, 2021.","title":"Balancing value underestimation and overestimation with realistic actor-critic","authors":"Sicen Li, Gang Wang, Qinyun Tang, Liquan Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2016), pages 4026-4034, Barcelona, Spain, 2016.","title":"Deep exploration via bootstrapped DQN","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":"Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2016)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pan et al., 2020] Ling Pan, Qingpeng Cai, and Longbo Huang. Softmax deep double deterministic policy gradi- ents. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2020), 2020.","title":"Softmax deep double deterministic policy gradients","authors":"Ling Pan, Qingpeng Cai, Longbo Huang","year":"2020","venue":"Proceedings of the 33rd Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pathak et al., 2019] Deepak Pathak, Dhiraj Gandhi, and Ab- hinav Gupta. Self-supervised exploration via disagree- ment. In Proceedings of the 36th International Conference on Machine Learning (ICML 2019), pages 5062-5071, Long Beach, California, USA, 2019.","title":"Self-supervised exploration via disagreement","authors":"Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning (ICML 2019)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learn- ing. In Proceedings of the 31st Conference on Neural In- formation Processing Systems (NIPS 2017), pages 2753- 2762, Long Beach, CA, USA, 2017.","title":"# exploration: A study of count-based exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2017","venue":"Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thrun and Schwartz, 1993] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for re- inforcement learning. In Proceedings of the 4th Connec- tionist Models Summer School, pages 255-263, 1993.","title":"Issues in using function approximation for reinforcement learning","authors":"Schwartz Thrun, Sebastian Thrun, Anton Schwartz","year":"1993","venue":"Proceedings of the 4th Connectionist Models Summer School","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[van Hasselt et al., 2016] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble Q-learning. In Proceedings of the 30th AAAI Confer- ence on Artificial Intelligence (AAAI 2016), pages 2094- 2100, Phoenix, Arizona, USA, 2016.","title":"Deep reinforcement learning with double Q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI 2016)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2017] Zongzhang Zhang, Zhiyuan Pan, and Mykel J Kochenderfer. Weighted double Q-learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI 2017), pages 3455-3461, Melbourne, Australia, 2017.","title":"Weighted double Q-learning","authors":"Zhang","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI 2017)","doi":null,"url":null,"internal_url":"/proceedings/2017/483"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}