{"type":"ScholarlyArticle","title":"Weighted Double Q-learning","authors":[{"name":"Zongzhang Zhang","slug":"zongzhang-zhang","given":"Zongzhang","family":"Zhang","affiliation":"Soochow University Suzhou , Jiangsu 215006 China"},{"name":"Zhiyuan Pan","slug":"zhiyuan-pan","given":"Zhiyuan","family":"Pan","affiliation":"Soochow University Suzhou , Jiangsu 215006 China"},{"name":"Mykel J. Kochenderfer","slug":"mykel-j-kochenderfer","given":"Mykel J.","family":"Kochenderfer","affiliation":"Stanford University Stanford , CA 94305 USA"}],"abstract":"Q-learning is a popular reinforcement learning algorithm, but it can perform poorly in stochastic environments due to overestimating action values. Overestimation is due to the use of a single estimator that uses the maximum action value as an approximation for the maximum expected action value. To avoid overestimation in Q-learning, the double Q-learning algorithm was recently proposed, which uses the double estimator method. It uses two estimators from independent sets of experiences, with one estimator determining the maximizing action and the other providing the estimate of its value. Double Q-learning sometimes underestimates the action values. This paper introduces a weighted double Q-learning algorithm, which is based on the construction of the weighted double estimator, with the goal of balancing between the overestimation in the single estimator and the underestimation in the double estimator. Empirically, the new algorithm is shown to perform well on several MDP problems.","doi":"10.24963/ijcai.2017/483","url":"https://ijcai.org/proceedings/2017/483","pdf":"https://ijcai.org/proceedings/2017/0483.pdf","markdown":"https://ijcai.org/proceedings/2017/483.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/483","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":483,"pages":{"first":3455,"last":3461},"keywords":["Uncertainty in AI: Markov Decision Processes","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Bellman, 1957] Richard Bellman. Dynamic Programming. Princeton University Press, 1957.","title":"Dynamic Programming","authors":"Richard Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 2007] Dimitri P. Bertsekas. Dynamic Program- ming and Optimal Control. Athena Scientific, Belmont, MA, 2007.","title":"Dynamic Programming and Optimal Control","authors":"Dimitri P Bertsekas","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brafman and Tennenholtz, 2001] Ronen I. Brafman and Moshe Tennenholtz. R-max -a general polynomial time algorithm for near-optimal reinforcement learning. In In- ternational Joint Conference on Artificial Intelligence (IJ- CAI), pages 953-958, 2001.","title":"R-max -a general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2001","venue":"ternational Joint Conference on Artificial Intelligence (IJ-CAI)","doi":null,"url":null,"internal_url":"/proceedings/2001/130"},{"position":4,"raw":"[D'Eramo et al., 2016] Carlo D'Eramo, Alessandro Nuara, and Marcello Restelli. Estimating the maximum expect- ed value through Gaussian approximation. In Internation- al Conference on Machine Learning (ICML), pages 1032- 1040, 2016.","title":"Estimating the maximum expected value through Gaussian approximation","authors":"Carlo D' Eramo, Alessandro Nuara, Marcello Restelli","year":"2016","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ernst et al., 2005] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learn- ing. Journal of Machine Learning Research, 6(1):503- 556, 2005.","title":"Tree-based batch mode reinforcement learning","authors":"Damien Ernst, Pierre Geurts, Louis Wehenkel","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kearns and Singh, 1999] Michael J. Kearns and Satinder P. Singh. Finite-sample convergence rates for Q-learning and indirect algorithms. In Advances in Neural Information Processing Systems (NIPS), pages 996-1002, 1999.","title":"Finite-sample convergence rates for Q-learning and indirect algorithms","authors":"J Michael, Kearns, P Satinder, Singh","year":"1999","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kochenderfer, 2015] Mykel J. Kochenderfer. Decision Making Under Uncertainty: Theory and Application. MIT Press, 2015.","title":"Decision Making Under Uncertainty: Theory and Application","authors":"J Mykel, Kochenderfer","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521:436-444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee and Powell, 2012] Donghun Lee and Warren B. Pow- ell. An intelligent battery controller using bias-corrected Q-learning. In AAAI Conference on Artificial Intelligence (AAAI), pages 316-322, 2012.","title":"Powell. An intelligent battery controller using bias-corrected Q-learning","authors":"Lee, Powell ;, Donghun Lee, Warren B","year":"2012","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2013] Donghun Lee, Boris Defourny, and War- ren B. Powell. Bias-corrected Q-learning to control max- operator bias in Q-learning. In IEEE Symposium on Adap- tive Dynamic Programming and Reinforcement Learning (ADPRL), pages 93-99, 2013.","title":"Donghun Lee, Boris Defourny, and Warren B. Powell. Bias-corrected Q-learning to control maxoperator bias in Q-learning","authors":"Lee","year":"2013","venue":"IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Littman, 2015] Michael L. Littman. Reinforcement learn- ing improves behaviour from evaluative feedback. Nature, 521:445-451, 2015.","title":"Reinforcement learning improves behaviour from evaluative feedback","authors":"L Michael, Littman","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Joel Veness Andrei A. Rusu, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Joel Veness, Andrei A Rusu, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Strehl et al., 2006] Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. PAC model-free reinforcement learning. In International Con- ference on Machine Learning (ICML), pages 881-888, 2006.","title":"PAC model-free reinforcement learning","authors":"Alexander L Strehl, Lihong Li, Eric Wiewiora, John Langford, Michael L Littman","year":"2006","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[van Hasselt et al., 2016] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble Q-learning. In AAAI Conference on Artificial Intelli- gence (AAAI), pages 2094-2010, 2016.","title":"Deep reinforcement learning with double Q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[van Hasselt, 2010] Hado van Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems (NIPS), pages 2613-2621, 2010.","title":"Hado van Hasselt. Double Q-learning","authors":"Van Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[van Hasselt, 2011] Hado van Hasselt. Insights in Reinforce- ment Learning. PhD thesis, Center for Mathematics and Computer Science, Utrecht University, 2011.","title":"Hado van Hasselt. Insights in Reinforcement Learning","authors":"Van Hasselt","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Watkins, 1989] Christopher J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, University of Cambridge, 1989.","title":"Learning from Delayed Rewards","authors":"J C H Christopher, Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wiering and van Otterlo, 2012] Marco Wiering and Martijn van Otterlo, editors. Reinforcement Learning: State of the Art. Springer, New York, 2012.","title":"Reinforcement Learning: State of the Art","authors":"Van Wiering, Otterlo","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}