{"type":"ScholarlyArticle","title":"Convergence Analysis of Gradient Descent for Eigenvector Computation","authors":[{"name":"Zhiqiang Xu","slug":"zhiqiang-xu","given":"Zhiqiang","family":"Xu","affiliation":"KAUST , Saudi Arabia"},{"name":"Xin Cao","slug":"xin-cao","given":"Xin","family":"Cao","affiliation":"UNSW , Australia"},{"name":"Xin Gao","slug":"xin-gao","given":"Xin","family":"Gao","affiliation":"KAUST , Saudi Arabia"}],"abstract":"We present a novel, simple and systematic convergence analysis of gradient descent for eigenvector computation. As a popular, practical, and provable approach to numerous machine learning problems, gradient descent has found successful applications to eigenvector computation as well. However, surprisingly, it lacks a thorough theoretical analysis for the underlying geodesically non-convex problem. In this work, the convergence of the gradient descent solver for the leading eigenvector computation is shown to be at a global rate O(min{ (lambda_1/Delta_p)^2 log(1/epsilon), 1/epsilon }), where Delta_p=lambda_p-lambda_p+1>0 represents the generalized positive eigengap and always exists without loss of generality with lambda_i being the i-th largest eigenvalue of the given real symmetric matrix and p being the multiplicity of lambda_1. The rate is linear at (lambda_1/Delta_p)^2 log(1/epsilon) if (lambda_1/Delta_p)^2=O(1), otherwise sub-linear at O(1/epsilon). We also show that the convergence only logarithmically instead of quadratically depends on the initial iterate. Particularly, this is the first time the linear convergence for the case that the conventionally considered eigengap Delta_1= lambda_1 - lambda_2=0 but the generalized eigengap Delta_p satisfies (lambda_1/Delta_p)^2=O(1), as well as the logarithmic dependence on the initial iterate are established for the gradient descent solver. We are also the first to leverage for analysis the log principal angle between the iterate and the space of globally optimal solutions. Theoretical properties are verified in experiments.","doi":"10.24963/ijcai.2018/407","url":"https://ijcai.org/proceedings/2018/407","pdf":"https://ijcai.org/proceedings/2018/0407.pdf","markdown":"https://ijcai.org/proceedings/2018/407.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/407","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":407,"pages":{"first":2933,"last":2939},"keywords":["Constraints and SAT: Constraint Optimisation","Heuristic Search and Game Playing: Combinatorial Search and Optimisation"],"videos":[],"references":[{"position":1,"raw":"[Absil et al., 2008] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2008.","title":"Optimization algorithms on matrix manifolds","authors":"P-A Absil, Robert Mahony, Rodolphe Sepulchre","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2012] Raman Arora, Andrew Cotter, Karen Livescu, and Nathan Srebro. Stochastic optimization for PCA and PLS. In 50th Annual Allerton Conference on Communication, Control, and Computing, Allerton, pages 861-868, 2012.","title":"Stochastic optimization for PCA and PLS","authors":"Raman Arora, Andrew Cotter, Karen Livescu, Nathan Srebro","year":"2012","venue":"50th Annual Allerton Conference on Communication, Control, and Computing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arora et al., 2013] Raman Arora, Andrew Cotter, and Nati Srebro. Stochastic optimization of PCA with capped MSG. In NIPS, pages 1815-1823, 2013.","title":"Stochastic optimization of PCA with capped MSG","authors":"Raman Arora, Andrew Cotter, Nati Srebro","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Balcan et al., 2016] Maria-Florina Balcan, Simon Shaolei Du, Yining Wang, and Adams Wei Yu. An improved gap- dependency analysis of the noisy power method. In COLT, pages 284-309, 2016.","title":"An improved gapdependency analysis of the noisy power method","authors":"Maria-Florina Balcan, Simon Shaolei Du, Yining Wang, Adams Wei Yu","year":"2016","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Balsubramani et al., 2013] Akshay Balsubramani, Sanjoy Dasgupta, and Yoav Freund. The fast convergence of in- cremental pca. In NIPS, pages 3174-3182.","title":"The fast convergence of incremental pca","authors":"Akshay Balsubramani, Sanjoy Dasgupta, Yoav Freund","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bonnabel, 2013] Silvere Bonnabel. Stochastic gradient de- scent on riemannian manifolds. IEEE Trans. Automat. Contr., 58(9):2217-2229, 2013.","title":"Stochastic gradient descent on riemannian manifolds","authors":"Silvere Bonnabel","year":"2013","venue":"IEEE Trans. Automat. Contr","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Edelman et al., 1999] Alan Edelman, Tomás A. Arias, and Steven T. Smith. The geometry of algorithms with or- thogonality constraints. SIAM J. Matrix Anal. Appl., 20(2):303-353, April 1999.","title":"The geometry of algorithms with orthogonality constraints","authors":"Alan Edelman, Tomás A Arias, Steven T Smith","year":"1999","venue":"SIAM J. Matrix Anal. Appl","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fan et al., 2018] Jianqing Fan, Qiang Sun, Wen-Xin Zhou, and Ziwei Zhu. Principal component analysis for big data. arXiv preprint arXiv:1801.01602, 2018.","title":"Principal component analysis for big data","authors":"Jianqing Fan, Qiang Sun, Wen-Xin Zhou, Ziwei Zhu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garber et al., 2016] Dan Garber, Elad Hazan, Chi Jin, Sham M. Kakade, Cameron Musco, Praneeth Netrapalli, and Aaron Sidford. Faster eigenvector computation via shift-and-invert preconditioning. In ICML, pages 2626- 2634, 2016.","title":"Faster eigenvector computation via shift-and-invert preconditioning","authors":"Dan Garber, Elad Hazan, Chi Jin, M Sham, Cameron Kakade, Praneeth Musco, Aaron Netrapalli, Sidford","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Golub and Van Loan, 1996] Gene H. Golub and Charles F. Van Loan. Matrix Computations (3rd Ed.). Johns Hopkins University Press, Baltimore, MD, USA, 1996.","title":"Matrix Computations","authors":"Gene H Golub, Charles F Van Loan","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hardt and Price, 2014] Moritz Hardt and Eric Price. The noisy power method: A meta algorithm with applications. In NIPS, pages 2861-2869, 2014.","title":"The noisy power method: A meta algorithm with applications","authors":"Moritz Hardt, Eric Price","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] Huikang Liu, Weijie Wu, and Anthony Man-Cho So. Quadratic optimization with orthogonality constraints: Explicit lojasiewicz exponent and linear con- vergence of line-search methods. In ICML, pages 1158- 1167, 2016.","title":"Quadratic optimization with orthogonality constraints: Explicit lojasiewicz exponent and linear convergence of line-search methods","authors":"Liu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Musco and Musco, 2015] Cameron Musco and Christopher Musco. Randomized block krylov methods for stronger and faster approximate singular value decomposition. In NIPS, pages 1396-1404, 2015.","title":"Randomized block krylov methods for stronger and faster approximate singular value decomposition","authors":"Musco Musco, Cameron Musco, Christopher Musco","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ng et al., 2002] Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In NIPS, pages 849-856. MIT Press, 2002.","title":"On spectral clustering: Analysis and an algorithm","authors":"Andrew Y Ng, Michael I Jordan, Yair Weiss","year":"2002","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pitaval et al., 2015] Renaud-Alexandre Pitaval, Wei Dai, and Olav Tirkkonen. Convergence of gradient descent for low-rank matrix approximation. IEEE Trans. Information Theory, 61(8):4451-4457, 2015.","title":"Convergence of gradient descent for low-rank matrix approximation","authors":"Renaud-Alexandre Pitaval, Wei Dai, Olav Tirkkonen","year":"2015","venue":"IEEE Trans. Information Theory","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rudelson and Vershynin, 2010] Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: extreme singular values. arXiv preprint arXiv:1003.2990, 2010.","title":"Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: extreme singular values","authors":"Vershynin Rudelson","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sa et al., 2015] Christopher De Sa, Christopher Re, and Kunle Olukotun. Global convergence of stochastic gra- dient descent for some non-convex matrix problems. In ICML, pages 2332-2341, 2015.","title":"Global convergence of stochastic gradient descent for some non-convex matrix problems","authors":"Christopher De Sa, Christopher Re, Kunle Olukotun","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shamir, 2015] Ohad Shamir. A stochastic PCA and SVD algorithm with an exponential convergence rate. In ICML, pages 144-152, 2015.","title":"A stochastic PCA and SVD algorithm with an exponential convergence rate","authors":"Ohad Shamir","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shamir, 2016a] Ohad Shamir. Convergence of stochastic gradient descent for PCA. In ICML, pages 257-265, 2016.","title":"Convergence of stochastic gradient descent for PCA","authors":"Shamir ; A] Ohad, Shamir","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shamir, 2016b] Ohad Shamir. Fast stochastic algorithms for SVD and PCA: convergence properties and convexity. In ICML, pages 248-256, 2016.","title":"Fast stochastic algorithms for SVD and PCA: convergence properties and convexity","authors":"Ohad Shamir, Shamir","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sra et al., 2011] Suvrit Sra, Sebastian Nowozin, and Stephen J. Wright. Optimization for Machine Learning. The MIT Press, 2011.","title":"Optimization for Machine Learning","authors":"Sra","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wen and Yin, 2013] Zaiwen Wen and Wotao Yin. A fea- sible method for optimization with orthogonality con- straints. Math. Program., 142(1-2):397-434, 2013.","title":"A feasible method for optimization with orthogonality constraints","authors":"Wen, Yin ;, Zaiwen Wen, Wotao Yin","year":"2013","venue":"Math. Program","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu and Gao, 2018] Zhiqiang Xu and Xin Gao. On truly block eigensolvers via riemannian optimization. In AIS- TATS, pages 168-177, 2018.","title":"On truly block eigensolvers via riemannian optimization","authors":"Zhiqiang Xu, Xin Gao","year":"2018","venue":"AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu and Ke, 2016a] Zhiqiang Xu and Yiping Ke. Effective and efficient spectral clustering on text and link data. In CIKM, pages 357-366, 2016.","title":"Effective and efficient spectral clustering on text and link data","authors":"Ke Xu, Zhiqiang Xu, Yiping Ke","year":"2016","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu and Ke, 2016b] Zhiqiang Xu and Yiping Ke. Stochas- tic variance reduced riemannian eigensolver. CoRR, abs/1605.08233, 2016.","title":"Stochastic variance reduced riemannian eigensolver","authors":"Ke Xu, Zhiqiang Xu, Yiping Ke","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2017] Zhiqiang Xu, Yiping Ke, and Xin Gao. A fast algorithm for matrix eigen-decompositionn. In UAI 2017, Sydney, Australia, August 11-15, 2017, 2017.","title":"A fast algorithm for matrix eigen-decompositionn","authors":"Zhiqiang Xu, Yiping Ke, Xin Gao","year":"2017","venue":"UAI 2017","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2016] Hongyi Zhang, Sashank J. Reddi, and Suvrit Sra. Riemannian SVRG: fast stochastic optimiza- tion on riemannian manifolds. In NIPS, pages 4592-4600, 2016.","title":"Riemannian SVRG: fast stochastic optimization on riemannian manifolds","authors":"Hongyi Zhang, J Sashank, Suvrit Reddi, Sra","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}