{"type":"ScholarlyArticle","title":"Adaptive Order Q-learning","authors":[{"name":"Tao Tan","slug":"tao-tan","given":"Tao","family":"Tan","affiliation":"College of Computer Science , Chongqing University"},{"name":"Hong Xie","slug":"hong-xie","given":"Hong","family":"Xie","affiliation":"University of Science and Technology of China"},{"name":"Defu Lian","slug":"defu-lian","given":"Defu","family":"Lian","affiliation":"University of Science and Technology of China"}],"abstract":"This paper revisits the estimation bias control problem of Q-learning, motivated by the fact that the estimation bias is not always evil, i.e., some environments benefit from overestimation bias or underestimation bias, while others suffer from these biases.  Different from previous coarse-grained bias control methods, this paper proposes a fine-grained bias control algorithm called Order Q-learning. It uses the order statistic of multiple independent Q-tables to control bias and flexibly meet the personalized bias needs of different environments, i.e., the bias can vary from underestimation bias to overestimation bias as one selects a higher order Q-value.  We derive the expected estimation bias and its lower bound and upper bound. They reveal that the expected estimation bias is inversely proportional to the number of Q-tables and proportional to the index of order statistic function. To show the versatility of Order Q-learning,  we design an adaptive parameter adjustment strategy, leading to AdaOrder (Adaptive Order) Q-learning. It adaptively selects the number of Q-tables and the index of order statistic function via the number of visits to state-action pair and the average Q-value. We extend Order Q-learning and AdaOrder Q-learning to the large scale setting with function approximation, leading to Order DQN and AdaOrder DQN, respectively. Finally, we consider two experiment settings: deep reinforcement learning experiments show that our method outperforms several SOTA baselines drastically; tabular MDP experiments reveal fundamental insights into why our method can achieve superior performance.Our supplementary file can be found in https://1drv.ms/f/s!Atddp1iaDmL2gjv31CaGquw5WwYI.","doi":"10.24963/ijcai.2024/547","url":"https://ijcai.org/proceedings/2024/547","pdf":"https://ijcai.org/proceedings/2024/0547.pdf","markdown":"https://ijcai.org/proceedings/2024/547.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/547","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":547,"pages":{"first":4946,"last":4954},"keywords":["Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Anschel et al., 2017] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabi- lization for deep reinforcement learning. In International conference on machine learning, pages 176-185. PMLR, 2017.","title":"Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning","authors":"Anschel","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bouten et al., 2005] Luc Bouten, Simon Edwards, and VP Belavkin. Bellman equations for optimal feedback control of qubit states. Journal of Physics B: Atomic, Molecular and Optical Physics, 38(3):151, 2005.","title":"Bellman equations for optimal feedback control of qubit states","authors":"Bouten","year":"2005","venue":"Journal of Physics B: Atomic, Molecular and Optical Physics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cetin and Celiktutan, 2023] Edoardo Cetin and Oya Celik- tutan. Learning pessimism for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pages 6971-6979, 2023.","title":"Learning pessimism for reinforcement learning","authors":"Edoardo Cetin, Oya Celiktutan","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without a model. In International Conference on Learning Representations, 2021.","title":"Randomized ensembled double qlearning: Learning fast without a model","authors":"Xinyue Chen, Che Wang, Zijian Zhou, Keith W Ross","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022] Liyu Chen, Rahul Jain, and Haipeng Luo. Learning infinite-horizon average-reward markov decision process with constraints. In International Conference on Machine Learning, pages 3246-3270. PMLR, 2022.","title":"Learning infinite-horizon average-reward markov decision process with constraints","authors":"Liyu Chen, Rahul Jain, Haipeng Luo","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2022] Chen Chu, Yong Li, Jinzhuo Liu, Shuyue Hu, Xuelong Li, and Zhen Wang. A formal model for mul- tiagent q-learning dynamics on regular graphs. In Luc De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vi- enna, Austria, 23-29 July 2022, pages 194-200. ijcai.org, 2022.","title":"A formal model for multiagent q-learning dynamics on regular graphs","authors":"Chen Chu, Yong Li, Jinzhuo Liu, Shuyue Hu, Xuelong Li, Zhen Wang","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022","doi":null,"url":null,"internal_url":"/proceedings/2022/28"},{"position":7,"raw":"[Ghasemipour et al., 2021] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.","title":"Emaq: Expected-max q-learning operator for simple yet effective offline and online rl","authors":"Seyed Kamyar, Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gong et al., 2023] Xiaoyu Gong, Shuai Lü, Jiayu Yu, Sheng Zhu, and Zongze Li. Adaptive estimation q-learning with uncertainty and familiarity. In Proceedings of the Thirty- Second International Joint Conference on Artificial Intel- ligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 3750-3758. ijcai.org, 2023. [Hasselt, 2010] Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23:2613-2621, 2010.","title":"Adaptive estimation q-learning with uncertainty and familiarity","authors":"Xiaoyu Gong, Shuai Lü, Jiayu Yu, Sheng Zhu, Zongze Li","year":"2010","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023","doi":null,"url":null,"internal_url":"/proceedings/2023/417"},{"position":9,"raw":"[Jones and Qin, 2022] Galin L Jones and Qian Qin. Markov chain monte carlo in practice. Annual Review of Statistics and Its Application, 9:557-578, 2022.","title":"Markov chain monte carlo in practice","authors":"L Galin, Qian Jones, Qin","year":"2022","venue":"Annual Review of Statistics and Its Application","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karimpanal et al., 2023] Thommen George Karimpanal, Hung Le, Majid Abdolshah, Santu Rana, Sunil Gupta, Truyen Tran, and Svetha Venkatesh. Balanced q-learning: Combining the influence of optimistic and pessimistic tar- gets. Artificial Intelligence, 325:104021, 2023.","title":"Balanced q-learning: Combining the influence of optimistic and pessimistic targets","authors":"Thommen George Karimpanal, Hung Le, Majid Abdolshah, Santu Rana, Sunil Gupta, Truyen Tran, Svetha Venkatesh","year":"2023","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kondrup et al., 2023] Flemming Kondrup, Thomas Jiraler- spong, Elaine Lau, Nathan de Lara, Jacob Shkrob, My Duc Tran, Doina Precup, and Sumana Basu. Towards safe me- chanical ventilation treatment using deep offline reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15696-15702, 2023.","title":"Towards safe mechanical ventilation treatment using deep offline reinforcement learning","authors":"Flemming Kondrup, Thomas Jiralerspong, Elaine Lau, Nathan De Lara, Jacob Shkrob, My Duc Tran, Doina Precup, Sumana Basu","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lan et al., 2020] Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Control- ling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.","title":"Maxmin q-learning: Controlling the estimation bias of q-learning","authors":"Qingfeng Lan, Yangchen Pan, Alona Fyshe, Martha White","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mannor et al., 2007] Shie Mannor, Duncan Simester, Peng Sun, and John N Tsitsiklis. Bias and variance approxi- mation in value function estimates. Management Science, 53(2):308-322, 2007.","title":"Bias and variance approximation in value function estimates","authors":"Shie Mannor, Duncan Simester, Peng Sun, John N Tsitsiklis","year":"2007","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mock and Muknahallipatna, 2023] James W Mock and Suresh S Muknahallipatna. A comparison of ppo, td3 and sac reinforcement algorithms for quadruped walking gait generation. Journal of Intelligent Learning Systems and Applications, 15(1):36-56, 2023.","title":"A comparison of ppo, td3 and sac reinforcement algorithms for quadruped walking gait generation","authors":"W James, Mock, S Suresh, Muknahallipatna","year":"2023","venue":"Journal of Intelligent Learning Systems and Applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peer et al., 2021] Oren Peer, Chen Tessler, Nadav Merlis, and Ron Meir. Ensemble bootstrapping for q-learning. In International Conference on Machine Learning, pages 8454-8463. PMLR, 2021.","title":"Ensemble bootstrapping for q-learning","authors":"Oren Peer, Chen Tessler, Nadav Merlis, Ron Meir","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pentaliotis and Wiering, 2021] Andreas Pentaliotis and Marco A Wiering. Variation-resistant q-learning: Con- trolling and utilizing estimation bias in reinforcement learning for better performance. In ICAART (2), pages 17-28, 2021.","title":"Variation-resistant q-learning: Controlling and utilizing estimation bias in reinforcement learning for better performance","authors":"Wiering Pentaliotis, Andreas Pentaliotis, Marco A Wiering","year":"2021","venue":"ICAART (2)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Song et al., 2019] Zhao Song, Ron Parr, and Lawrence Carin. Revisiting the softmax bellman operator: New ben- efits and new perspective. In International Conference on Machine Learning, pages 5916-5925. PMLR, 2019.","title":"Revisiting the softmax bellman operator: New benefits and new perspective","authors":"Zhao Song, Ron Parr, Lawrence Carin","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018. [Tesauro and others, 1995]","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58-68, 1995.","title":"Temporal difference learning and td-gammon","authors":"Gerald Tesauro","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Thrun and Schwartz, 1993] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for rein- forcement learning. In Proceedings of the Fourth Connec- tionist Models Summer School, pages 255-263.","title":"Issues in using function approximation for reinforcement learning","authors":"Schwartz Thrun, Sebastian Thrun, Anton Schwartz","year":"1993","venue":"Proceedings of the Fourth Connectionist Models Summer School","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Upadhyay, 2021] Indrima Upadhyay. ANALYSIS OF Q- LEARNING BASED GAME PLAYING AGENTS FOR AB- STRACT BOARD GAMES WITH INCREASING STATE- SPACE COMPLEXITY. PhD thesis, Miami University, 2021.","title":"ANALYSIS OF Q-LEARNING BASED GAME PLAYING AGENTS FOR AB-STRACT BOARD GAMES WITH INCREASING STATE-SPACE COMPLEXITY","authors":"Indrima Upadhyay","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Urtans and Nikitenko, 2018] Evalds Urtans and Agris Nikitenko. Survey of deep q-network variants in pygame learning environment. In Proceedings of the 2018 2nd International Conference on Deep Learning Technologies, pages 27-36, 2018.","title":"Evalds Urtans and Agris Nikitenko. Survey of deep q-network variants in pygame learning environment","authors":"Nikitenko Urtans","year":"2018","venue":"Proceedings of the 2018 2nd International Conference on Deep Learning Technologies","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Van Hasselt, 2013] Hado Van Hasselt. Estimating the max- imum expected value: an analysis of (nested) cross vali- dation and the maximum sample average. arXiv preprint arXiv:1302.7175, 2013.","title":"Hado Van Hasselt. Estimating the maximum expected value: an analysis of (nested) cross validation and the maximum sample average","authors":"Van Hasselt","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Hang Wang, Sen Lin, and Junshan Zhang. Adaptive ensemble q-learning: Minimizing esti- mation bias via error feedback. Advances in Neural Infor- mation Processing Systems, 34, 2021.","title":"Adaptive ensemble q-learning: Minimizing estimation bias via error feedback","authors":"Wang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Watkins, 1989] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.","title":"Learning from delayed rewards","authors":"Christopher John, Cornish Hellaby Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Williams et al., 2017] Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1714-1721. IEEE, 2017.","title":"Information theoretic mpc for model-based reinforcement learning","authors":"Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, Evangelos A Theodorou","year":"2017","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2022] Haolin Wu, Jianwei Zhang, Zhuang Wang, Yi Lin, and Hui Li. Sub-avg: Overestimation reduction for cooperative multi-agent reinforcement learning. Neu- rocomputing, 474:94-106, 2022.","title":"Sub-avg: Overestimation reduction for cooperative multi-agent reinforcement learning","authors":"Haolin Wu, Jianwei Zhang, Zhuang Wang, Yi Lin, Hui Li","year":"2022","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2020] Yongjian Yang, Xintao Wang, Yuanbo Xu, and Qiuyang Huang. Multiagent reinforcement learning-based taxi predispatching model to balance taxi supply and demand. Journal of Advanced Transportation, 2020, 2020.","title":"Multiagent reinforcement learning-based taxi predispatching model to balance taxi supply and demand","authors":"Yongjian Yang, Xintao Wang, Yuanbo Xu, Qiuyang Huang","year":"2020","venue":"Journal of Advanced Transportation","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Young and Tian, 2019] Kenny Young and Tian Tian. Mi- natar: An atari-inspired testbed for more efficient reinforcement learning experiments. arXiv preprint arXiv:1903.03176, 59:60, 2019.","title":"Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments","authors":"Kenny Young, Tian Tian","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2017] Zongzhang Zhang, Zhiyuan Pan, and Mykel J Kochenderfer. Weighted double q-learning. In IJCAI, pages 3455-3461, 2017.","title":"Weighted double q-learning","authors":"Zhang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/483"},{"position":33,"raw":"[Zhang et al., 2023] Zhe Zhang, Yukun Zou, Junjie Lai, and Qing Xu. M2DQN: A robust method for accelerating deep q-learning network. In Proceedings of the 15th Inter- national Conference on Machine Learning and Comput- ing, ICMLC 2023, Zhuhai, China, February 17-20, 2023, pages 116-120. ACM, 2023.","title":"M2DQN: A robust method for accelerating deep q-learning network","authors":"Zhe Zhang, Yukun Zou, Junjie Lai, Qing Xu","year":"2023","venue":"Proceedings of the 15th International Conference on Machine Learning and Computing, ICMLC 2023","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2023] Fuqing Zhao, Qiaoyun Wang, and Ling Wang. An inverse reinforcement learning framework with the q-learning mechanism for the metaheuristic algorithm. Knowledge-Based Systems, 265:110368, 2023.","title":"An inverse reinforcement learning framework with the q-learning mechanism for the metaheuristic algorithm","authors":"Zhao","year":"2023","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu and Rigotti, 2021] Rong Zhu and Mattia Rigotti. Self- correcting q-learning. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pages 11185- 11192, 2021.","title":"Selfcorrecting q-learning","authors":"Rigotti ; ] Rong Zhu, Mattia Zhu, Rigotti","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}