{"type":"ScholarlyArticle","title":"Exploring the Inefficiency of Heavy Ball as Momentum Parameter Approaches 1","authors":[{"name":"Xiaoge Deng","slug":"xiaoge-deng","given":"Xiaoge","family":"Deng","affiliation":"College of Computer Science and Technology , National University of Defense Technology , China"},{"name":"Tao Sun","slug":"tao-sun","given":"Tao","family":"Sun","affiliation":"College of Computer Science and Technology , National University of Defense Technology , China"},{"name":"Dongsheng Li","slug":"dongsheng-li","given":"Dongsheng","family":"Li","affiliation":"College of Computer Science and Technology , National University of Defense Technology , China"},{"name":"Xicheng Lu","slug":"xicheng-lu","given":"Xicheng","family":"Lu","affiliation":"College of Computer Science and Technology , National University of Defense Technology , China"}],"abstract":"The heavy ball momentum method is a commonly used technique for accelerating training processes in the machine learning community. However, empirical evidence suggests that the convergence of stochastic gradient descent (SGD) with heavy ball may slow down when the momentum hyperparameter approaches 1. Despite this observation, there are no established theories or solutions to explain and address this issue. In this study, we provide the first theoretical result that elucidates why momentum slows down SGD as it tends to 1. To better understand this inefficiency, we focus on the quadratic convex objective in the analysis. Our findings show that momentum accelerates SGD when the scaling parameter is not very close to 1. Conversely, when the scaling parameter approaches 1, momentum impairs SGD and degrades its stability. Based on the theoretical findings, we propose a descending warmup technique for the heavy ball momentum, which exploits the advantages of the heavy ball method and overcomes the inefficiency problem when the momentum tends to 1. Numerical results demonstrate the effectiveness of the proposed SHB-DW algorithm.","doi":"10.24963/ijcai.2024/431","url":"https://ijcai.org/proceedings/2024/431","pdf":"https://ijcai.org/proceedings/2024/0431.pdf","markdown":"https://ijcai.org/proceedings/2024/431.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/431","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":431,"pages":{"first":3899,"last":3907},"keywords":["Machine Learning: ML: Optimization","Machine Learning: ML: Applications","Machine Learning: ML: Learning theory"],"videos":[],"references":[{"position":1,"raw":"[Aybat et al., 2020] Necdet Serhat Aybat, Alireza Fallah, Mert Gurbuzbalaban, and Asuman Ozdaglar. Robust ac- celerated gradient methods for smooth strongly convex functions. SIAM Journal on Optimization, 30(1):717-751, 2020.","title":"Robust accelerated gradient methods for smooth strongly convex functions","authors":"Aybat","year":"2020","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bousquet and Elisseeff, 2002] Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of machine learning research, 2(Mar):499-526, 2002.","title":"Stability and generalization","authors":"Elisseeff Bousquet, Olivier Bousquet, André Elisseeff","year":"2002","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Yuansi Chen, Chi Jin, and Bin Yu. Sta- bility and convergence trade-off of iterative optimization algorithms. arXiv preprint arXiv:1804.01619, 2018.","title":"Stability and convergence trade-off of iterative optimization algorithms","authors":"Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ganesh et al., 2023] Swetha Ganesh, Rohan Deb, Gugan Thoppe, and Amarjit Budhiraja. Does momentum help in stochastic optimization? A sample complexity analy- sis. In Robin J. Evans and Ilya Shpitser, editors, Pro- ceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 602-612. PMLR, 31 Jul-04 Aug 2023.","title":"Does momentum help in stochastic optimization? A sample complexity analysis","authors":"Swetha Ganesh, Rohan Deb, Gugan Thoppe, Amarjit Budhiraja","year":"2023","venue":"Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghadimi et al., 2015] Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global conver- gence of the heavy-ball method for convex optimization. In European control conference, pages 310-315, 2015.","title":"Global convergence of the heavy-ball method for convex optimization","authors":"Ghadimi","year":"2015","venue":"European control conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gitman et al., 2019] Igor Gitman, Hunter Lang, Pengchuan Zhang, and Lin Xiao. Understanding the role of mo- mentum in stochastic gradient methods. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32.","title":"Understanding the role of momentum in stochastic gradient methods","authors":"Igor Gitman, Hunter Lang, Pengchuan Zhang, Lin Xiao","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630-645. Springer, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jastrze ¸bski et al., 2018] Stanislaw Jastrze ¸bski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. Workshop of ICLR, 2018.","title":"Three factors influencing minima in SGD","authors":"Jastrze ¸bski","year":"2018","venue":"Workshop of ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Cur- ran Associates, Inc., 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Keskar and Socher, 2017] Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from Adam to SGD. arXiv preprint arXiv:1712.07628, 2017.","title":"Improving generalization performance by switching from Adam to SGD","authors":"Nitish Shirish, Keskar, Richard Socher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kidambi et al., 2018] Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of existing momentum schemes for stochastic optimization. In 2018 Information Theory and Applications Workshop (ITA), pages 1-9. IEEE, 2018.","title":"On the insufficiency of existing momentum schemes for stochastic optimization","authors":"Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, Sham Kakade","year":"2018","venue":"2018 Information Theory and Applications Workshop (ITA)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Con- ference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kulunchakov and Mairal, 2019] Andrei Kulunchakov and Julien Mairal. A generic acceleration framework for stochastic composite optimization. In Advances in Neu- ral Information Processing Systems, pages 12556-12567, 2019.","title":"A generic acceleration framework for stochastic composite optimization","authors":"Mairal Kulunchakov, Andrei Kulunchakov, Julien Mairal","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun and Cortes, 2010] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.","title":"MNIST handwritten digit database","authors":"Lecun, Cortes, Yann Lecun, Corinna Cortes","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu and Belkin, 2020] Chaoyue Liu and Mikhail Belkin. Accelerating SGD with momentum for over- parameterized learning. In International Conference on Learning Representations, 2020.","title":"Accelerating SGD with momentum for overparameterized learning","authors":"Belkin Liu, Chaoyue Liu, Mikhail Belkin","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma and Yarats, 2019] Jerry Ma and Denis Yarats. Quasi- hyperbolic momentum and Adam for deep learning. In International Conference on Learning Representations, 2019.","title":"Quasihyperbolic momentum and Adam for deep learning","authors":"Jerry Ma, Denis Yarats","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nesterov, 1983] Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o(1/k 2 ). In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.","title":"A method for solving the convex programming problem with convergence rate o(1/k 2 )","authors":"E Yurii, Nesterov","year":"1983","venue":"Dokl. akad. nauk Sssr","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nesterov, 2005] Yu Nesterov. Smooth minimization of non- smooth functions. Mathematical programming, 103:127- 152, 2005.","title":"Smooth minimization of nonsmooth functions","authors":"Yu Nesterov","year":"2005","venue":"Mathematical programming","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nesterov, 2013] Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.","title":"Introductory lectures on convex optimization: A basic course","authors":"Yurii Nesterov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ochs et al., 2014] Peter Ochs, Yunjin Chen, Thomas Brox, and Thomas Pock. ipiano: Inertial proximal algorithm for nonconvex optimization. SIAM Journal on Imaging Sci- ences, 7(2):1388-1419, 2014.","title":"ipiano: Inertial proximal algorithm for nonconvex optimization","authors":"Peter Ochs, Yunjin Chen, Thomas Brox, Thomas Pock","year":"2014","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ong, 2017] Ming Yang Ong. Understanding generalization. PhD thesis, Massachusetts Institute of Technology, 2017.","title":"Understanding generalization","authors":"Ming Yang, Ong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Polyak, 1964] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4(5):1-17, 1964.","title":"Some methods of speeding up the convergence of iteration methods","authors":"T Boris, Polyak","year":"1964","venue":"USSR Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rakhlin et al., 2012] Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Inter- national Conference on Machine Learning, pages 1571- 1578, 2012.","title":"Making gradient descent optimal for strongly convex stochastic optimization","authors":"Alexander Rakhlin, Ohad Shamir, Karthik Sridharan","year":"2012","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ramezani-Kebrya et al., 2024] Ali Ramezani-Kebrya, Ki- mon Antonakopoulos, Volkan Cevher, Ashish Khisti, and Ben Liang. On the generalization of stochastic gradient descent with momentum. Journal of Machine Learning Research, 25(22):1-56, 2024.","title":"On the generalization of stochastic gradient descent with momentum","authors":"Ali Ramezani-Kebrya, Kimon Antonakopoulos, Volkan Cevher, Ashish Khisti, Ben Liang","year":"2024","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.","title":"The annals of mathematical statistics","authors":"Monro Robbins, Herbert Robbins, Sutton Monro","year":"1951","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Stephan et al., 2017] Mandt Stephan, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approx- imate bayesian inference. Journal of Machine Learning Research, 18(134):1-35, 2017.","title":"Stochastic gradient descent as approximate bayesian inference","authors":"Mandt Stephan, Matthew D Hoffman, David M Blei","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2019a] Tao Sun, Dongsheng Li, Zhe Quan, Hao Jiang, Shengguo Li, and Yong Dou. Heavy-ball algorithms always escape saddle points. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelli- gence, IJCAI 2019, pages 3520-3526. ijcai.org, 2019.","title":"Heavy-ball algorithms always escape saddle points","authors":"Sun","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019","doi":null,"url":null,"internal_url":"/proceedings/2019/488"},{"position":32,"raw":"[Sun et al., 2019b] Tao Sun, Yuejiao Sun, Dongsheng Li, and Qing Liao. General proximal incremental aggregated gradient algorithms: Better and novel results under gen- eral scheme. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32.","title":"General proximal incremental aggregated gradient algorithms: Better and novel results under general scheme","authors":"Sun","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sun et al., 2019c] Tao Sun, Penghang Yin, Dongsheng Li, Chun Huang, Lei Guan, and Hao Jiang. Non-ergodic con- vergence analysis of heavy-ball algorithms. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 5033-5040, 2019.","title":"Non-ergodic convergence analysis of heavy-ball algorithms","authors":"Sun","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sun et al., 2020] Tao Sun, Linbo Qiao, and Dongsheng Li. Nonergodic complexity of proximal inertial gradient de- scents. IEEE Transactions on Neural Networks and Learn- ing Systems, 32(10):4613-4626, 2020.","title":"Nonergodic complexity of proximal inertial gradient descents","authors":"Tao Sun, Linbo Qiao, Dongsheng Li","year":"2020","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Sun et al., 2021] Tao Sun, Huaming Ling, Zuoqiang Shi, Dongsheng Li, and Bao Wang. Training deep neural net- works with adaptive momentum inspired by the quadratic optimization. arXiv preprint arXiv:2110.09057, 2021.","title":"Training deep neural networks with adaptive momentum inspired by the quadratic optimization","authors":"Tao Sun, Huaming Ling, Zuoqiang Shi, Dongsheng Li, Bao Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Vaswani et al., 2019] Sharan Vaswani, Francis Bach, and Mark Schmidt. Fast and faster convergence of SGD for over-parameterized models and an accelerated perceptron. In The 22nd International Conference on Artificial Intelli- gence and Statistics, pages 1195-1204. PMLR, 2019.","title":"Fast and faster convergence of SGD for over-parameterized models and an accelerated perceptron","authors":"Sharan Vaswani, Francis Bach, Mark Schmidt","year":"2019","venue":"The 22nd International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2022] Bao Wang, Tan M. Nguyen, Tao Sun, Andrea L. Bertozzi, Richard G. Baraniuk, and Stan- ley J. Osher. Scheduled restart momentum for acceler- ated stochastic gradient descent. SIAM J. Imaging Sci., 15(2):738-761, 2022.","title":"Scheduled restart momentum for accelerated stochastic gradient descent","authors":"Bao Wang, Tan M Nguyen, Tao Sun, Andrea L Bertozzi, Richard G Baraniuk, Stanley J Osher","year":"2022","venue":"SIAM J. Imaging Sci","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Welling and Teh, 2011] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynam- ics. In Proceedings of the 28th international conference on machine learning, pages 681-688.","title":"Bayesian learning via stochastic gradient langevin dynamics","authors":"Teh ; ] Max Welling, Yee W Welling, Teh","year":"2011","venue":"Proceedings of the 28th international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wiegerinck et al., 1994] Wim Wiegerinck, Andrzej Ko- moda, and Tom Heskes. Stochastic dynamics of learning with momentum in neural networks. Journal of Physics A: Mathematical and General, 27(13):4425, 1994.","title":"Stochastic dynamics of learning with momentum in neural networks","authors":"Wiegerinck","year":"1994","venue":"Journal of Physics A: Mathematical and General","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yan et al., 2018] Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic mo- mentum methods for deep learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artifi- cial Intelligence, pages 2955-2961, 7 2018.","title":"A unified analysis of stochastic momentum methods for deep learning","authors":"Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, Yi Yang","year":null,"venue":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/410"},{"position":43,"raw":"[Yuan et al., 2016] Kun Yuan, Bicheng Ying, and Ali H. Sayed. On the influence of momentum acceleration on online learning. Journal of Machine Learning Research, 17(192):1-66, 2016.","title":"On the influence of momentum acceleration on online learning","authors":"Kun Yuan, Bicheng Ying, Ali H Sayed","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}