{"type":"ScholarlyArticle","title":"Provable Acceleration of Nesterov’s Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks","authors":[{"name":"Xin Liu","slug":"xin-liu","given":"Xin","family":"Liu","affiliation":"Army Engineering University of PLA"},{"name":"Wei Tao","slug":"wei-tao","given":"Wei","family":"Tao","affiliation":"Strategic Assessments and Consultation Institute , Academy of Military Science"},{"name":"Wei Li","slug":"wei-li","given":"Wei","family":"Li","affiliation":"Army Engineering University of PLA"},{"name":"Dazhi Zhan","slug":"dazhi-zhan","given":"Dazhi","family":"Zhan","affiliation":"Army Engineering University of PLA"},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"Army Engineering University of PLA"},{"name":"Zhisong Pan","slug":"zhisong-pan","given":"Zhisong","family":"Pan","affiliation":"Army Engineering University of PLA"}],"abstract":"Due to its simplicity and efficiency, the first-order gradient method has been extensively employed in training neural networks. Although the optimization problem of the neural network is non-convex, recent research has proved that the first-order method is capable of attaining a global minimum during training over-parameterized neural networks, where the number of parameters is significantly larger than that of training instances. Momentum methods, including the heavy ball (HB) method and Nesterov's accelerated gradient (NAG) method, are the workhorse of first-order gradient methods owning to their accelerated convergence. In practice, NAG often exhibits superior performance than HB. However, current theoretical works fail to distinguish their convergence difference in training neural networks. To fill this gap, we consider the training problem of the two-layer ReLU neural network under over-parameterization and random initialization. Leveraging high-resolution dynamical systems and neural tangent kernel (NTK) theory, our result not only establishes tighter upper bounds of the convergence rate for both HB and NAG, but also provides the first theoretical guarantee for the acceleration of NAG over HB in training neural networks. Finally, we validate our theoretical results on three benchmark datasets.","doi":"10.24963/ijcai.2024/508","url":"https://ijcai.org/proceedings/2024/508","pdf":"https://ijcai.org/proceedings/2024/0508.pdf","markdown":"https://ijcai.org/proceedings/2024/508.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/508","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":508,"pages":{"first":4596,"last":4604},"keywords":["Machine Learning: ML: Optimization","Machine Learning: ML: Theory of deep learning"],"videos":[],"references":[{"position":1,"raw":"[Alemohammad et al., 2020] Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The re- current neural tangent kernel. In International Conference on Learning Representations, 2020.","title":"The recurrent neural tangent kernel","authors":"Alemohammad","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2019a] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, pages 8141-8150, 2019.","title":"On exact computation with an infinitely wide neural net","authors":"Arora","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arora et al., 2019b] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332, 2019.","title":"Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks","authors":"Arora","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bradbury et al., 2018] James Bradbury, Roy Frostig, Pe- ter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: Composable transformations of Python+NumPy programs, 2018.","title":"JAX: Composable transformations of Python+NumPy programs","authors":"James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vanderplas, Skye Wanderman-Milne, Qiao Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Sys- tems, pages 1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bu et al., 2021] Zhiqi Bu, Shiyun Xu, and Kan Chen. A dynamical view on optimization algorithms of overparam- eterized neural networks. In International Conference on Artificial Intelligence and Statistics, pages 3187-3195, 2021.","title":"A dynamical view on optimization algorithms of overparameterized neural networks","authors":"Bu","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Du et al., 2019a] Simon S. Du, Kangcheng Hou, Ruslan Salakhutdinov, Barnabás Póczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. In Advances in Neural Information Processing Systems, pages 5724-5734, 2019.","title":"Graph neural tangent kernel: Fusing graph neural networks with graph kernels","authors":"Du","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Du et al., 2019b] Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. International Conference on Machine Learning, pages 1675-1685, 2019.","title":"Gradient descent finds global minima of deep neural networks","authors":"Du","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Du et al., 2019c] Simon S. Du, Xiyu Zhai, Barnabás Póczos, and Aarti Singh. Gradient descent provably op- timizes over-parameterized neural networks. In Interna- tional Conference on Learning Representations, 2019.","title":"Gradient descent provably optimizes over-parameterized neural networks","authors":"Du","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghadimi et al., 2015] Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global conver- gence of the heavy-ball method for convex optimization. In European Control Conference, pages 310-315, 2015.","title":"Global convergence of the heavy-ball method for convex optimization","authors":"Ghadimi","year":"2015","venue":"European Control Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2020] Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks gen- eralize better than deep feedforward networks -A neural tangent kernel perspective. In Advances in Neural Infor- mation Processing Systems, pages 2698-2709, 2020.","title":"Why do deep residual networks generalize better than deep feedforward networks -A neural tangent kernel perspective","authors":"Kaixuan Huang, Yuqing Wang, Molei Tao, Tuo Zhao","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jacot et al., 2018] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing System, pages 8571-8580, 2018.","title":"Neural tangent kernel: Convergence and generalization in neural networks","authors":"Arthur Jacot, Franck Gabriel, Clément Hongler","year":"2018","venue":"Advances in Neural Information Processing System","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Laborde and Oberman, 2020] Maxime Laborde and Adam M. Oberman. A lyapunov analysis for accelerated gradient methods: from deterministic to stochastic case. In International Conference on Artificial Intelligence and Statistics, pages 602-612, 2020.","title":"A lyapunov analysis for accelerated gradient methods: from deterministic to stochastic case","authors":"Maxime Laborde, Adam M Oberman","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lessard et al., 2016] Laurent Lessard, Benjamin Recht, and Andrew Packard. Analysis and design of optimization al- gorithms via integral quadratic constraints. SIAM Journal on Optimization, 26(1):57-95, 2016.","title":"Analysis and design of optimization algorithms via integral quadratic constraints","authors":"Laurent Lessard, Benjamin Recht, Andrew Packard","year":"2016","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2019] Zhiyuan Li, Ruosong Wang, Dingli Yu, Si- mon S Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora. Enhanced convolutional neural tangent kernels. arXiv preprint arXiv:1911.00809, 2019.","title":"Enhanced convolutional neural tangent kernels","authors":"Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, Sanjeev Arora","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2022a] Jinlan Liu, Jun Kong, Dongpo Xu, Miao Qi, and Yinghua Lu. Convergence analysis of adabound with relaxed bound functions for non-convex optimization. Neural Networks, 145:300-307, 2022.","title":"Convergence analysis of adabound with relaxed bound functions for non-convex optimization","authors":"Liu","year":"2022","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2022b] Xin Liu, Zhisong Pan, and Wei Tao. Provable convergence of nesterov's accelerated gradi- ent method for over-parameterized neural networks. Knowledge-Based Systems, 251:109277, 2022.","title":"Provable convergence of nesterov's accelerated gradient method for over-parameterized neural networks","authors":"Liu","year":"2022","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nado et al., 2021] Zachary Nado, Justin M Gilmer, Christo- pher J Shallue, Rohan Anil, and George E Dahl. A large batch optimizer reality check: Traditional, generic optimizers suffice across batch sizes. arXiv preprint arXiv:2102.06356, 2021.","title":"A large batch optimizer reality check: Traditional, generic optimizers suffice across batch sizes","authors":"Zachary Nado, Justin M Gilmer, Christopher J Shallue, Rohan Anil, George E Dahl","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nakakita, 2022] Shogo Nakakita. Parametric estimation of stochastic differential equations via online gradient de- scent. arXiv preprint arXiv:2210.08800, 2022.","title":"Parametric estimation of stochastic differential equations via online gradient descent","authors":"Shogo Nakakita","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nesterov, 1983] Yurii Nesterov. A method of solving a con- vex programming problem with convergence rate o (1/k2). In Sov. Math. Dokl, 1983.","title":"A method of solving a convex programming problem with convergence rate o (1/k2)","authors":"Yurii Nesterov","year":"1983","venue":"Sov. Math. Dokl","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nesterov, 2013] Yurii Nesterov. Introductory lectures on convex optimization: A basic course. Springer Science, 2013.","title":"Introductory lectures on convex optimization: A basic course","authors":"Yurii Nesterov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nguyen, 2021] Quynh Nguyen. On the proof of global con- vergence of gradient descent for deep relu networks with linear widths. In International Conference on Machine Learning, pages 8056-8062, 2021.","title":"On the proof of global convergence of gradient descent for deep relu networks with linear widths","authors":"Quynh Nguyen","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Polyak, 1964] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4(5):1-17, 1964.","title":"Some methods of speeding up the convergence of iteration methods","authors":"T Boris, Polyak","year":"1964","venue":"USSR Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schmidhuber, 2015] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85- 117, 2015.","title":"Deep learning in neural networks: An overview","authors":"Jürgen Schmidhuber","year":"2015","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schmidt et al., 2021] Robin M. Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded val- ley -benchmarking deep learning optimizers. In Inter- national Conference on Machine Learning, pages 9367- 9376, 2021.","title":"Descending through a crowded valley -benchmarking deep learning optimizers","authors":"Robin M Schmidt, Frank Schneider, Philipp Hennig","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shi et al., 2021] Bin Shi, Simon S Du, Michael I Jordan, and Weijie J Su. Understanding the acceleration phe- nomenon via high-resolution differential equations. Math- ematical Programming, pages 1-70, 2021.","title":"Understanding the acceleration phenomenon via high-resolution differential equations","authors":"Bin Shi, Simon S Du, Michael I Jordan, Weijie J Su","year":"2021","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Song and Yang, 2019] Zhao Song and Xin Yang. Quadratic suffices for over-parametrization via matrix chernoff bound. arXiv preprint arXiv:1906.03593, 2019.","title":"Quadratic suffices for over-parametrization via matrix chernoff bound","authors":"Zhao Song, Xin Yang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Su et al., 2014] Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. In Ad- vances in Neural Information Processing Systems, pages 2510-2518, 2014.","title":"A differential equation for modeling nesterov's accelerated gradient method: Theory and insights","authors":"Weijie Su, Stephen Boyd, Emmanuel Candes","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2020] Boya Sun, Jemin George, and Solmaz S. Kia. High-resolution modeling of the fastest first-order op- timization method for strongly convex functions. In Con- ference on Decision and Control, pages 4237-4242, 2020.","title":"High-resolution modeling of the fastest first-order optimization method for strongly convex functions","authors":"Boya Sun, Jemin George, Solmaz S Kia","year":"2020","venue":"Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2021] Jun-Kun Wang, Chi-Heng Lin, and Ja- cob D. Abernethy. A modular analysis of provable ac- celeration via polyak's momentum: Training a wide relu network and a deep linear network. In International Con- ference on Machine Learning, pages 10816-10827, 2021.","title":"A modular analysis of provable acceleration via polyak's momentum: Training a wide relu network and a deep linear network","authors":"Wang","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wilson et al., 2021] Ashia C Wilson, Ben Recht, and Michael I Jordan. A lyapunov analysis of accelerated methods in optimization. Journal of Machine Learning Research, 22:1-34, 2021.","title":"A lyapunov analysis of accelerated methods in optimization","authors":"Ben Ashia C Wilson, Michael I Recht, Jordan","year":"2021","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2021] Dongpo Xu, Shengdong Zhang, Huisheng Zhang, and Danilo P. Mandic. Convergence of the rm- sprop deep learning method with penalty for nonconvex optimization. Neural Networks, 139:17-23, 2021.","title":"Convergence of the rmsprop deep learning method with penalty for nonconvex optimization","authors":"Xu","year":"2021","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2021] Peiyuan Zhang, Antonio Orvieto, Hadi Daneshmand, Thomas Hofmann, and Roy S. Smith. Re- visiting the role of euler numerical integration on acceler- ation and stability in convex optimization. In International Conference on Artificial Intelligence and Statistics, pages 3979-3987, 2021.","title":"Revisiting the role of euler numerical integration on acceleration and stability in convex optimization","authors":"Zhang","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}