{"type":"ScholarlyArticle","title":"Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent","authors":[{"name":"Bowen Weng","slug":"bowen-weng","given":"Bowen","family":"Weng","affiliation":"Electrical and Computer Engineering , The Ohio State University , Columbus , OH , USA."},{"name":"Huaqing Xiong","slug":"huaqing-xiong","given":"Huaqing","family":"Xiong","affiliation":"Electrical and Computer Engineering , The Ohio State University , Columbus , OH , USA."},{"name":"Yingbin Liang","slug":"yingbin-liang","given":"Yingbin","family":"Liang","affiliation":"Electrical and Computer Engineering , The Ohio State University , Columbus , OH , USA."},{"name":"Wei Zhang","slug":"wei-zhang","given":"Wei","family":"Zhang","affiliation":"Mechanical and Energy Engineering , Southern University of Science and Technology , China."}],"abstract":"Existing convergence analyses of Q-learning mostly focus on the vanilla stochastic gradient descent (SGD) type of updates. Despite the Adaptive Moment Estimation (Adam) has been commonly used for practical Q-learning algorithms, there has not been any convergence guarantee provided for Q-learning with such type of updates. In this paper, we first characterize the convergence rate for Q-AMSGrad, which is the Q-learning algorithm with AMSGrad update (a commonly adopted alternative of Adam for theoretical analysis). To further improve the performance, we propose to incorporate the momentum restart scheme to Q-AMSGrad, resulting in the so-called Q-AMSGradR algorithm. The convergence rate of Q-AMSGradR is also established. Our experiments on a linear quadratic regulator problem demonstrate that the two proposed Q-learning algorithms outperform the vanilla Q-learning with SGD updates. The two algorithms also exhibit significantly better performance than the DQN learning method over a batch of Atari 2600 games.","doi":"10.24963/ijcai.2020/422","url":"https://ijcai.org/proceedings/2020/422","pdf":"https://ijcai.org/proceedings/2020/0422.pdf","markdown":"https://ijcai.org/proceedings/2020/422.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/422","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":422,"pages":{"first":3051,"last":3057},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Deep Learning","Machine Learning: Reinforcement Learning","Constraints and SAT: Constraint Optimization"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26702/file_26702.m3u8","length_sec":613,"provider":"hls"}],"references":[{"position":1,"raw":"[Bertsekas and Tsitsiklis, 1996] Dimitri P. Bertsekas and John N Tsitsiklis. Neuro-Dynamic Programming, volume 5. Athena Scientific, 1996.","title":"Neuro-Dynamic Programming","authors":"Dimitri P Bertsekas, John N Tsitsiklis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 1995] Dimitri P Bertsekas. Dynamic program- ming and optimal control, volume 1. 1995.","title":"Dynamic programming and optimal control","authors":"Dimitri P Bertsekas","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bhandari et al., 2018] Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal dif- ference learning with linear function approximation. In Conference on Learning Theory, 2018.","title":"A finite time analysis of temporal difference learning with linear function approximation","authors":"Jalaj Bhandari, Daniel Russo, Raghav Singal","year":"2018","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019a] Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of Adam-type algorithms for non-convex optimization. In International Conference on Learning Representations, 2019.","title":"On the convergence of a class of Adam-type algorithms for non-convex optimization","authors":"Chen","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2019b] Zaiwei Chen, Sheng Zhang, Thinh T. Doan, Siva Theja Maguluri, and John-Paul Clarke. Finite- time analysis of Q-learning with linear function approxima- tion. arXiv preprint arXiv:1905.11425, 2019.","title":"Finitetime analysis of Q-learning with linear function approximation","authors":"Chen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Du et al., 2019] Simon S Du, Yuping Luo, Ruosong Wang, and Hanrui Zhang. Provably efficient Q-learning with func- tion approximation via distribution shift error checking oracle. In Advances in Neural Information Processing Sys- tems, pages 8058-8068, 2019.","title":"Provably efficient Q-learning with function approximation via distribution shift error checking oracle","authors":"Yuping Simon S Du, Ruosong Luo, Hanrui Wang, Zhang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Even-Dar and Mansour, 2003] Eyal Even-Dar and Yishay Mansour. Learning rates for Q-learning. Journal of Ma- chine Learning Research, 5:1-25, Dec 2003.","title":"Learning rates for Q-learning","authors":"Eyal Even, -Dar, Yishay Mansour","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Knight and Lerner, 2018] Ethan Knight and Osher Lerner. Natural gradient deep Q-learning. arXiv preprint arXiv:1803.07482, 2018.","title":"Natural gradient deep Q-learning","authors":"Lerner Knight, Ethan Knight, Osher Lerner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lewis and Vrabie, 2009] F. L. Lewis and D. Vrabie. Rein- forcement learning and adaptive dynamic programming for feedback control. IEEE Circuits and Systems Magazine, 9(3):32-50, Third 2009.","title":"Reinforcement learning and adaptive dynamic programming for feedback control","authors":"F L Lewis, D Vrabie","year":"2009","venue":"IEEE Circuits and Systems Magazine","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2018] Tyler Lu, Dale Schuurmans, and Craig Boutilier. Non-delusional Q-learning and value-iteration. In Proceedings of the Thirty-second Conference on Neu- ral Information Processing Systems (NeurIPS-18), pages 9971-9981, Montreal, QC, 2018.","title":"Non-delusional Q-learning and value-iteration","authors":"Tyler Lu, Dale Schuurmans, Craig Boutilier","year":"2018","venue":"Proceedings of the Thirty-second Conference on Neural Information Processing Systems (NeurIPS-18)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational Conference on Machine Learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Mnih","year":"2016","venue":"ternational Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[O'donoghue and Candes, 2015] Brendan O'donoghue and Emmanuel Candes. Adaptive restart for accelerated gradi- ent schemes. Foundations of computational mathematics, 15(3):715-732, 2015.","title":"Brendan O'donoghue and Emmanuel Candes. Adaptive restart for accelerated gradient schemes","authors":"O' Donoghue, Candes","year":"2015","venue":"Foundations of computational mathematics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Reddi et al., 2018] Sashank J. Reddi, Satyen Kale, and San- jiv Kumar. On the convergence of Adam and beyond. In In- ternational Conference on Learning Representations, 2018.","title":"On the convergence of Adam and beyond","authors":"J Sashank, Satyen Reddi, Sanjiv Kale, Kumar","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tran and others, 2019] Phuong Thi Tran et al. On the conver- gence proof of AMSGrad and a new version. IEEE Access, 7:61706-61716, 2019.","title":"On the convergence proof of AMSGrad and a new version","authors":"Phuong Thi, Tran","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble Q-learning. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.","title":"Deep reinforcement learning with double Q-learning","authors":"Van Hasselt","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learn- ing. In International Conference on Machine Learning, pages 1995-2003, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, Nando De Freitas","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Watkins and Dayan, 1992] Christopher J.C.H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiong et al., 2020] Huaqing Xiong, Tengyu Xu, Yingbin Liang, and Wei Zhang. Non-asymptotic convergence of Adam-type reinforcement learning algorithms under marko- vian sampling. arXiv preprint arXiv:2002.06286, 2020.","title":"Non-asymptotic convergence of Adam-type reinforcement learning algorithms under markovian sampling","authors":"Huaqing Xiong, Tengyu Xu, Yingbin Liang, Wei Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu and Gu, 2019] Pan Xu and Quanquan Gu. A finite-time analysis of q-learning with neural network function approx- imation. arXiv preprint arXiv:1912.04511, 2019.","title":"A finite-time analysis of q-learning with neural network function approximation","authors":"Gu Xu, Pan Xu, Quanquan Gu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2019] Zhuora Yang, Yuchen Xie, and Zhaoran Wang. A theoretical analysis of deep Q-learning. arXiv preprint arXiv:1901.00137, 2019.","title":"A theoretical analysis of deep Q-learning","authors":"Yang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2018] Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. On the convergence of adap- tive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671, 2018.","title":"On the convergence of adaptive gradient methods for nonconvex optimization","authors":"Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zou et al., 2019a] Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for convergences of adam and rmsprop. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11127-11135, 2019.","title":"A sufficient condition for convergences of adam and rmsprop","authors":"Zou","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zou et al., 2019b] Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear func- tion approximation. In Advances in Neural Information Processing Systems, pages 8665-8675, 2019.","title":"Finite-sample analysis for sarsa with linear function approximation","authors":"Zou","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}