{"type":"ScholarlyArticle","title":"Heavy-ball Algorithms Always Escape Saddle Points","authors":[{"name":"Tao Sun","slug":"tao-sun","given":"Tao","family":"Sun","affiliation":"Computer of College , National University of Defense Technology"},{"name":"Dongsheng Li","slug":"dongsheng-li","given":"Dongsheng","family":"Li","affiliation":"Computer of College , National University of Defense Technology"},{"name":"Zhe Quan","slug":"zhe-quan","given":"Zhe","family":"Quan","affiliation":"Hunan University"},{"name":"Hao Jiang","slug":"hao-jiang","given":"Hao","family":"Jiang","affiliation":"Computer of College , National University of Defense Technology"},{"name":"Shengguo Li","slug":"shengguo-li","given":"Shengguo","family":"Li","affiliation":"Computer of College , National University of Defense Technology"},{"name":"Yong Dou","slug":"yong-dou","given":"Yong","family":"Dou","affiliation":"Computer of College , National University of Defense Technology"}],"abstract":"Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoid saddle points? The answer is yes! Direct using the existing proof technique for the heavy-ball algorithms is hard due to that each iteration of the heavy-ball algorithm consists of current and last points. It is impossible to formulate the algorithms as iteration like xk+1= g(xk) under some mapping g. To this end, we design a new mapping on a new space. With some transfers, the heavy-ball algorithm can be interpreted as iterations after this mapping. Theoretically, we prove that heavy-ball gradient descent enjoys larger stepsize than the gradient descent to escape saddle points to escape the saddle point. And the heavy-ball proximal point algorithm is also considered; we also proved that the algorithm can always escape the saddle point.","doi":"10.24963/ijcai.2019/488","url":"https://ijcai.org/proceedings/2019/488","pdf":"https://ijcai.org/proceedings/2019/0488.pdf","markdown":"https://ijcai.org/proceedings/2019/488.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/488","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":488,"pages":{"first":3520,"last":3526},"keywords":["Machine Learning: Learning Theory","Machine Learning: Probabilistic Machine Learning","Heuristic Search and Game Playing: Combinatorial Search and Optimisation","Machine Learning: Explainable Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Alvarez and Attouch, 2001] Felipe Alvarez and Hedy At- touch. An inertial proximal method for maximal mono- tone operators via discretization of a nonlinear oscillator with damping. Set-Valued Analysis, 9(1-2):3-11, 2001.","title":"Felipe Alvarez and Hedy Attouch. An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping","authors":"Attouch Alvarez","year":"2001","venue":"Set-Valued Analysis","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alvarez, 2000] Felipe Alvarez. On the minimizing proper- ty of a second order dissipative system in hilbert spaces. SIAM Journal on Control and Optimization, 38(4):1102- 1119, 2000.","title":"On the minimizing property of a second order dissipative system in hilbert spaces","authors":"Felipe Alvarez","year":"2000","venue":"SIAM Journal on Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arora et al., 2015] Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. 2015.","title":"Simple, efficient, and neural algorithms for sparse coding","authors":"Sanjeev Arora, Rong Ge, Tengyu Ma, Ankur Moitra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bolte et al., 2014] Jérôme Bolte, Shoham Sabach, and Mar- c Teboulle. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Mathematical Programming, 146(1-2):459-494, 2014.","title":"Proximal alternating linearized minimization for nonconvex and nonsmooth problems","authors":"Jérôme Bolte, Shoham Sabach, Marc Teboulle","year":"2014","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cai et al., 2016] T Tony Cai, Xiaodong Li, and Zongming Ma. Optimal rates of convergence for noisy sparse phase retrieval via thresholded wirtinger flow. The Annals of S- tatistics, 44(5):2221-2251, 2016.","title":"Optimal rates of convergence for noisy sparse phase retrieval via thresholded wirtinger flow","authors":"Tony Cai, Xiaodong Li, Zongming Ma","year":"2016","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaudhari et al., 2016] Pratik Chaudhari, Anna Choroman- ska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccar- do Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.","title":"Entropy-sgd: Biasing gradient descent into wide valleys","authors":"Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann Lecun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, Riccardo Zecchina","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Combettes and Glaudin, 2017] Patrick L. Combettes and Lilian E. Glaudin. Quasinonexpansive iterations on the affine hull of orbits: From mann's mean value algorithm to inertial methods. Siam Journal on Optimization, 27(4), 2017.","title":"Quasinonexpansive iterations on the affine hull of orbits: From mann's mean value algorithm to inertial methods","authors":"Patrick L Combettes, Lilian E Glaudin","year":"2017","venue":"Siam Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ge et al., 2015] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gra- dient for tensor decomposition. In Conference on Learning Theory, pages 797-842, 2015.","title":"Escaping from saddle points-online stochastic gradient for tensor decomposition","authors":"Rong Ge, Furong Huang, Chi Jin, Yang Yuan","year":"2015","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ge et al., 2016] Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. In Ad- vances in Neural Information Processing Systems, pages 2973-2981, 2016.","title":"Matrix completion has no spurious local minimum","authors":"Rong Ge, Jason D Lee, Tengyu Ma","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghadimi et al., 2015] Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global conver- gence of the heavy-ball method for convex optimization. In Control Conference (ECC), 2015 European, pages 310-315. IEEE, 2015.","title":"Global convergence of the heavy-ball method for convex optimization","authors":"Ghadimi","year":"2015","venue":"Control Conference (ECC), 2015 European","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2017a] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In Proceedings of the 34th In- ternational Conference on Machine Learning-Volume 70, pages 1724-1732. JMLR. org, 2017.","title":"How to escape saddle points efficiently","authors":"Jin","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2017b] Chi Jin, Praneeth Netrapalli, and Michael I Jordan. Accelerated gradient descent es- capes saddle points faster than gradient descent. arXiv preprint arXiv:1711.10456, 2017.","title":"Accelerated gradient descent escapes saddle points faster than gradient descent","authors":"Jin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Keshavan et al., 2010] Raghunandan H Keshavan, Andrea Montanari, and Sewoong Oh. Matrix completion from a few entries. IEEE transactions on information theory, 56(6):2980-2998, 2010.","title":"Matrix completion from a few entries","authors":"Andrea Raghunandan H Keshavan, Sewoong Montanari, Oh","year":"2010","venue":"IEEE transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kurdyka, 1998] Krzysztof Kurdyka. On gradients of func- tions definable in o-minimal structures. In Annales de l'institut Fourier, volume 48, pages 769-784. Chartres: L'Institut, 1950-, 1998.","title":"On gradients of functions definable in o-minimal structures","authors":"Krzysztof Kurdyka","year":"1950","venue":"Annales de l'institut Fourier","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lange, 2013] Kenneth Lange. Elementary optimization. In Optimization, pages 1-21. Springer, 2013.","title":"Elementary optimization","authors":"Kenneth Lange","year":"2013","venue":"Optimization","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2016] Jason D Lee, Max Simchowitz, Michael I Jordan, and Benjamin Recht. Gradient descent only con- verges to minimizers. In Conference on Learning Theory, pages 1246-1257, 2016.","title":"Gradient descent only converges to minimizers","authors":"Jason D Lee, Max Simchowitz, Benjamin Michael I Jordan, Recht","year":"2016","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee et al., 2017] Jason D Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael I Jordan, and Ben- jamin Recht. First-order methods almost always avoid sad- dle points. To be appeared in Mathematical Progamming, 2017.","title":"First-order methods almost always avoid saddle points","authors":"Lee","year":"2017","venue":"Mathematical Progamming","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Loizou and Richtárik, 2017a] Nicolas Loizou and Peter Richtárik. Linearly convergent stochastic heavy ball method for minimizing generalization error. arXiv preprint arXiv:1710.10737, 2017.","title":"Linearly convergent stochastic heavy ball method for minimizing generalization error","authors":"Richtárik Loizou, Nicolas Loizou, Peter Richtárik","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Loizou and Richtárik, 2017b] Nicolas Loizou and Peter Richtárik. Momentum and stochastic momentum for s- tochastic gradient, newton, proximal point and subspace descent methods. arXiv preprint arXiv:1712.09677, 2017.","title":"Momentum and stochastic momentum for stochastic gradient, newton, proximal point and subspace descent methods","authors":"Richtárik Loizou, Nicolas Loizou, Peter Richtárik","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Łojasiewicz, 1993] Stanislas Łojasiewicz. Sur la géométrie semi-et sous-analytique. Ann. Inst. Fourier, 43(5):1575- 1595, 1993.","title":"Sur la géométrie semi-et sous-analytique","authors":"Stanislas Łojasiewicz","year":"1993","venue":"Ann. Inst. Fourier","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Moudafi and Elizabeth, 2003] Abdellatif Moudafi and E Elizabeth. Approximate inertial proximal methods using the enlargement of maximal monotone operators. International Journal of Pure and Applied Mathematics, 5(3):283-299, 2003.","title":"Approximate inertial proximal methods using the enlargement of maximal monotone operators","authors":"Abdellatif Moudafi, Elizabeth","year":"2003","venue":"International Journal of Pure and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ochs et al., 2014] Peter Ochs, Yunjin Chen, Thomas Brox, and Thomas Pock. ipiano: Inertial proximal algorithm for nonconvex optimization. SIAM Journal on Imaging Sci- ences, 7(2):1388-1419, 2014.","title":"ipiano: Inertial proximal algorithm for nonconvex optimization","authors":"Peter Ochs, Yunjin Chen, Thomas Brox, Thomas Pock","year":"2014","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pemantle, 1990] Robin Pemantle. Nonconvergence to un- stable points in urn models and stochastic approximations. The Annals of Probability, 18(2):698-712, 1990.","title":"Robin Pemantle. Nonconvergence to unstable points in urn models and stochastic approximations","authors":"Pemantle","year":"1990","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rockafellar and Wets, 2009] R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.","title":"Variational analysis","authors":"Rockafellar Tyrrell, J-B Roger, Wets","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shub, 2013] Michael Shub. Global stability of dynamical systems. Springer Science & Business Media, 2013.","title":"Global stability of dynamical systems","authors":"Michael Shub","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2017] Ju Sun, Qing Qu, and John Wright. Com- plete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2):853-884, 2017.","title":"Complete dictionary recovery over the sphere i: Overview and the geometric picture","authors":"Ju Sun, Qing Qu, John Wright","year":"2017","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2019] Tao Sun, Penghang Yin, Dongsheng Li, Chun Huang, Lei Guan, and Hao Jiang. Non-ergodic con- vergence analysis of heavy-ball algorithms. AAAI, 2019.","title":"Non-ergodic convergence analysis of heavy-ball algorithms","authors":"Tao Sun, Penghang Yin, Dongsheng Li, Chun Huang, Lei Guan, Hao Jiang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu and Yin, 2013] Yangyang Xu and Wotao Yin. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor fac- torization and completion. SIAM Journal on imaging sci- ences, 6(3):1758-1789, 2013.","title":"A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion","authors":"Yin Xu, Yangyang Xu, Wotao Yin","year":"2013","venue":"SIAM Journal on imaging sciences","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zavriev and Kostyuk, 1993] SK Zavriev and FV Kostyuk. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling, 4(4):336-341, 1993.","title":"Heavy-ball method in nonconvex optimization problems","authors":"Kostyuk ; Sk Zavriev, Zavriev, Kostyuk","year":"1993","venue":"Computational Mathematics and Modeling","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}