{"type":"ScholarlyArticle","title":"A Unified Analysis of Stochastic Momentum Methods for Deep Learning","authors":[{"name":"Yan Yan","slug":"yan-yan","given":"Yan","family":"Yan","affiliation":"SUSTech-UTS Joint Centre of CIS , Southern University of Science and Technology; Centre for Artificial Intelligence , University of Technology Sydney"},{"name":"Tianbao Yang","slug":"tianbao-yang","given":"Tianbao","family":"Yang","affiliation":"Department of Computer Science , The University of Iowa"},{"name":"Zhe Li","slug":"zhe-li","given":"Zhe","family":"Li","affiliation":"Department of Computer Science , The University of Iowa"},{"name":"Qihang Lin","slug":"qihang-lin","given":"Qihang","family":"Lin","affiliation":"Tippie College of Business , The University of Iowa"},{"name":"Yi Yang","slug":"yi-yang","given":"Yi","family":"Yang","affiliation":"SUSTech-UTS Joint Centre of CIS , Southern University of Science and Technology; Centre for Artificial Intelligence , University of Technology Sydney"}],"abstract":"Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochastic gradient (SG) method, and the stochastic momentum methods including two famous variants, i.e., the stochastic heavy-ball (SHB) method and the stochastic variant of Nesterov?s accelerated gradient (SNAG) method. We propose a framework that unifies the three variants. We then derive the convergence rates of the norm of gradient for the non-convex optimization problem, and analyze the generalization performance through the uniform stability approach. Particularly, the convergence analysis of the training objective exhibits that SHB and SNAG have no advantage over SG. However, the stability analysis shows that the momentum term can improve the stability of the learned model and hence improve the generalization performance. These theoretical insights verify the common wisdom and are also corroborated by our empirical analysis on deep learning.","doi":"10.24963/ijcai.2018/410","url":"https://ijcai.org/proceedings/2018/410","pdf":"https://ijcai.org/proceedings/2018/0410.pdf","markdown":"https://ijcai.org/proceedings/2018/410.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/410","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":410,"pages":{"first":2955,"last":2961},"keywords":["Machine Learning: Neural Networks","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Bousquet and Elisseeff, 2002] Olivier Bousquet and André Elisseeff. Stability and generalization. J. Mach. Learn. Res., 2:499-526, 2002.","title":"Stability and generalization","authors":"Elisseeff Bousquet, Olivier Bousquet, André Elisseeff","year":"2002","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cesa-Bianchi et al., 2004] Nicolò Cesa-Bianchi, Alex Con- coni, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Trans. on Inf. Theory, 50(9), 2004.","title":"On the generalization ability of on-line learning algorithms","authors":"Cesa-Bianchi","year":"2004","venue":"IEEE Trans. on Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ghadimi and Lan, 2013] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for non- convex stochastic programming. SIAM Journal on Opti- mization, 23(4):2341-2368, 2013.","title":"Stochastic first-and zeroth-order methods for nonconvex stochastic programming","authors":"Saeed Ghadimi, Guanghui Lan","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghadimi and Lan, 2016] Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlin- ear and stochastic programming. Math. Program., 156(1- 2):59-99, 2016.","title":"Accelerated gradient methods for nonconvex nonlinear and stochastic programming","authors":"Saeed Ghadimi, Guanghui Lan","year":"2016","venue":"Math. Program","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghadimi et al., 2014] Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global conver- gence of the heavy-ball method for convex optimization. coRR, 2014.","title":"Global convergence of the heavy-ball method for convex optimization","authors":"Ghadimi","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hardt et al., 2016] Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the Interna- tional Conference on Machine Learning, 2016.","title":"Train faster, generalize better: Stability of stochastic gradient descent","authors":"Moritz Hardt, Benjamin Recht, Yoram Singer","year":"2016","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hillar and Lim, 2013] Christopher J. Hillar and Lek-Heng Lim. Most tensor problems are np-hard. J. ACM, 60(6):45:1-45:39, 2013.","title":"Most tensor problems are np-hard","authors":"Christopher J Hillar, Lek-Heng Lim","year":"2013","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2012] Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury. Deep neural net- works for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29:82-97, 2012.","title":"Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups","authors":"Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-Rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, Brian Kingsbury","year":"2012","venue":"Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin and Rosasco, 2016] Junhong Lin and Lorenzo Rosasco. Optimal learning for multi-pass stochastic gradient meth- ods. In NIPS, pages 4556-4564, 2016.","title":"Optimal learning for multi-pass stochastic gradient methods","authors":"Lin, Rosasco ;, Junhong Lin, Lorenzo Rosasco","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nesterov, 1983] Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/sqr(k)). Soviet Mathematics Doklady, 27:372-376, 1983.","title":"A method of solving a convex programming problem with convergence rate O(1/sqr(k))","authors":"Yurii Nesterov","year":"1983","venue":"Soviet Mathematics Doklady","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nesterov, 2004] Yurii Nesterov. Introductory lectures on convex optimization : a basic course. Applied optimiza- tion. Kluwer Academic Publ., 2004.","title":"Introductory lectures on convex optimization : a basic course","authors":"Yurii Nesterov","year":"2004","venue":"Applied optimization","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ochs et al., 2014] Peter Ochs, Yunjin Chen, Thomas Brox, and Thomas Pock. ipiano: Inertial proximal algorithm for nonconvex optimization. SIAM J. Imaging Sciences, 7(2):1388-1419, 2014.","title":"ipiano: Inertial proximal algorithm for nonconvex optimization","authors":"Peter Ochs, Yunjin Chen, Thomas Brox, Thomas Pock","year":"2014","venue":"SIAM J. Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ochs et al., 2015] Peter Ochs, Thomas Brox, and Thomas Pock. ipiasco: Inertial proximal algorithm for strongly convex optimization. Journal of Mathematical Imaging and Vision, 53(2):171-181, 2015.","title":"ipiasco: Inertial proximal algorithm for strongly convex optimization","authors":"Ochs","year":"2015","venue":"Journal of Mathematical Imaging and Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ochs, 2016] Peter Ochs. Local convergence of the heavy- ball method and ipiano for non-convex optimization. CoRR, abs/1606.09070, 2016.","title":"Local convergence of the heavyball method and ipiano for non-convex optimization","authors":"Peter Ochs","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Polyak, 1964] B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Compu- tational Mathematics and Mathematical Physics, 4:791- 803, 1964.","title":"Some methods of speeding up the convergence of iteration methods","authors":"B T Polyak","year":"1964","venue":"USSR Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Reddi et al., 2016] Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alexander J. Smola. Stochas- tic variance reduction for nonconvex optimization. CoRR, abs/1603.06160, 2016.","title":"Stochastic variance reduction for nonconvex optimization","authors":"J Sashank, Ahmed Reddi, Suvrit Hefny, Barnabás Sra, Alexander J Póczos, Smola","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the impor- tance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 1139-1147, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Ilya Sutskever, James Martens, George E Dahl, Geoffrey E Hinton","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhu and Hazan, 2016] Zeyuan Allen Zhu and Elad Hazan. Variance reduction for faster non-convex optimization. CoRR, abs/1603.05643, 2016.","title":"Variance reduction for faster non-convex optimization","authors":"Hazan ; ] Zhu, Allen Zeyuan, Elad Zhu, Hazan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}