{"type":"ScholarlyArticle","title":"Tight Convergence Rate of Gradient Descent for Eigenvalue Computation","authors":[{"name":"Qinghua Ding","slug":"qinghua-ding","given":"Qinghua","family":"Ding","affiliation":"Department of Computer Science and Engineering , The Chineses University of Hong Kong"},{"name":"Kaiwen Zhou","slug":"kaiwen-zhou","given":"Kaiwen","family":"Zhou","affiliation":"Department of Computer Science and Engineering , The Chineses University of Hong Kong"},{"name":"James Cheng","slug":"james-cheng","given":"James","family":"Cheng","affiliation":"Department of Computer Science and Engineering , The Chineses University of Hong Kong"}],"abstract":"Riemannian gradient descent (RGD) is a simple, popular and efficient algorithm for leading eigenvector computation [AMS08]. However, the existing analysis of RGD for eigenproblem is still not tight, which is O(log(n/epsilon)/Delta^2) due to [Xu et al., 2018]. In this paper, we show that RGD in fact converges at rate O(log(n/epsilon)/Delta), and give instances to shows the tightness of our result. This improves the best prior analysis by a quadratic factor. Besides, we also give tight convergence analysis of a deterministic variant of Oja's rule due to [Oja, 1982]. We show that it also enjoys fast convergence rate of O(log(n/epsilon)/Delta). Previous papers only gave asymptotic characterizations [Oja, 1982; Oja, 1989; Yi et al., 2005]. Our tools for proving convergence results include an innovative reduction and chaining technique, and a noisy fixed point iteration argument. Besides, we also give empirical justifications of our convergence rates over synthetic and real data.","doi":"10.24963/ijcai.2020/453","url":"https://ijcai.org/proceedings/2020/453","pdf":"https://ijcai.org/proceedings/2020/0453.pdf","markdown":"https://ijcai.org/proceedings/2020/453.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/453","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":453,"pages":{"first":3276,"last":3282},"keywords":["Machine Learning: Dimensionality Reduction and Manifold Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25270/file_25270.m3u8","length_sec":286,"provider":"hls"}],"references":[{"position":1,"raw":"[Absil et al., 2009] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2009.","title":"Optimization algorithms on matrix manifolds","authors":"P-A Absil, Robert Mahony, Rodolphe Sepulchre","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amari, 1998] Shun-Ichi Amari. Natural gradient works ef- ficiently in learning. Neural computation, 10(2):251-276, 1998.","title":"Shun-Ichi Amari. Natural gradient works efficiently in learning","authors":"Amari","year":"1998","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boumal et al., 2014] Nicolas Boumal, Bamdev Mishra, P-A Absil, and Rodolphe Sepulchre. Manopt, a matlab toolbox for optimization on manifolds. The Journal of Machine Learning Research, 15(1):1455-1459, 2014.","title":"Manopt, a matlab toolbox for optimization on manifolds","authors":"Nicolas Boumal, Bamdev Mishra, P-A Absil, Rodolphe Sepulchre","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 1998] Tianping Chen, Yingbo Hua, and Wei- Yong Yan. Global convergence of oja's subspace algorithm for principal component extraction. IEEE Transactions on Neural Networks, 9(1):58-67, 1998.","title":"Global convergence of oja's subspace algorithm for principal component extraction","authors":"Tianping Chen, Yingbo Hua, Wei-Yong Yan","year":"1998","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hairer et al., 2006] Ernst Hairer, Christian Lubich, and Ger- hard Wanner. Geometric numerical integration: structure- preserving algorithms for ordinary differential equations, volume 31. Springer Science & Business Media, 2006.","title":"and Gerhard Wanner","authors":"Hairer","year":"2006","venue":"Geometric numerical integration: structurepreserving algorithms for ordinary differential equations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Helmke and Moore, 2012] Uwe Helmke and John B Moore. Optimization and dynamical systems. Springer Science & Business Media, 2012.","title":"Optimization and dynamical systems","authors":"Moore ; Helmke, Uwe Helmke, John B Moore","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2018] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Pro- cessing Systems, pages 6389-6399, 2018.","title":"Visualizing the loss landscape of neural nets","authors":"Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Musco and Musco, 2015] Cameron Musco and Christopher Musco. Randomized block krylov methods for stronger and faster approximate singular value decomposition. In Advances in Neural Information Processing Systems, pages 1396-1404, 2015.","title":"Randomized block krylov methods for stronger and faster approximate singular value decomposition","authors":"Musco Musco, Cameron Musco, Christopher Musco","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Oja, 1982] Erkki Oja. Simplified neuron model as a princi- pal component analyzer. Journal of mathematical biology, 15(3):267-273, 1982.","title":"Simplified neuron model as a principal component analyzer","authors":"Erkki Oja, Oja","year":"1982","venue":"Journal of mathematical biology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Oja, 1989] Erkki Oja. Neural networks, principal compo- nents, and subspaces. International journal of neural sys- tems, 1(01):61-68, 1989.","title":"Neural networks, principal components, and subspaces","authors":"Erkki Oja, Oja","year":"1989","venue":"International journal of neural systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Pascanu and Bengio, 2013] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013.","title":"Revisiting natural gradient for deep networks","authors":"Razvan Pascanu, Yoshua Bengio","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Raskutti and Mukherjee, 2015] Garvesh Raskutti and Sayan Mukherjee. The information geometry of mirror descent. IEEE Transactions on Information Theory, 61(3):1451- 1457, 2015.","title":"Garvesh Raskutti and Sayan Mukherjee. The information geometry of mirror descent","authors":"Mukherjee Raskutti","year":"2015","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shamir, 2015] Ohad Shamir. A stochastic pca and svd al- gorithm with an exponential convergence rate. In Interna- tional Conference on Machine Learning, pages 144-152, 2015.","title":"A stochastic pca and svd algorithm with an exponential convergence rate","authors":"Ohad Shamir","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wen and Yin, 2013] Zaiwen Wen and Wotao Yin. A fea- sible method for optimization with orthogonality con- straints. Mathematical Programming, 142(1-2):397-434, 2013.","title":"A feasible method for optimization with orthogonality constraints","authors":"Wen, Yin ;, Zaiwen Wen, Wotao Yin","year":"2013","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Xu et al., 2018] Zhiqiang Xu, Xin Cao, and Xin Gao. Con- vergence analysis of gradient descent for eigenvector com- putation. International Joint Conferences on Artificial In- telligence Organization, 2018.","title":"Convergence analysis of gradient descent for eigenvector computation","authors":"Zhiqiang Xu, Xin Cao, Xin Gao","year":"2018","venue":"International Joint Conferences on Artificial Intelligence Organization","doi":null,"url":null,"internal_url":"/proceedings/2018/407"},{"position":16,"raw":"[Yan et al., 1994] Wei-Yong Yan, Uwe Helmke, and John B Moore. Global analysis of oja's flow for neural networks. IEEE Transactions on Neural Networks, 5(5):674-683, 1994.","title":"Global analysis of oja's flow for neural networks","authors":"Wei-Yong Yan, Uwe Helmke, John B Moore","year":"1994","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yi et al., 2005] Zhang Yi, Mao Ye, Jian Cheng Lv, and Kok Kiong Tan. Convergence analysis of a determinis- tic discrete time system of oja's pca learning algorithm. IEEE Transactions on Neural Networks, 16(6):1318- 1328, 2005.","title":"Convergence analysis of a deterministic discrete time system of oja's pca learning algorithm","authors":"Zhang Yi, Mao Ye, Jian Cheng Lv, Kok Kiong, Tan","year":"2005","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang and Sra, 2016] Hongyi Zhang and Suvrit Sra. First- order methods for geodesically convex optimization. In Conference on Learning Theory, pages 1617-1638, 2016.","title":"Firstorder methods for geodesically convex optimization","authors":"Sra Zhang, Hongyi Zhang, Suvrit Sra","year":"2016","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang and Sra, 2018] Hongyi Zhang and Suvrit Sra. An es- timate sequence for geodesically convex optimization. In Conference On Learning Theory, pages 1703-1723, 2018.","title":"An estimate sequence for geodesically convex optimization","authors":"Sra Zhang, Hongyi Zhang, Suvrit Sra","year":"2018","venue":"Conference On Learning Theory","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}