{"type":"ScholarlyArticle","title":"SVD-free Convex-Concave Approaches for Nuclear Norm Regularization","authors":[{"name":"Yichi Xiao","slug":"yichi-xiao","given":"Yichi","family":"Xiao","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Zhe Li","slug":"zhe-li","given":"Zhe","family":"Li","affiliation":"Department of Computer Science , the University of Iowa , Iowa City , IA 52242 , USA"},{"name":"Tianbao Yang","slug":"tianbao-yang","given":"Tianbao","family":"Yang","affiliation":"Department of Computer Science , the University of Iowa , Iowa City , IA 52242 , USA"},{"name":"Lijun Zhang","slug":"lijun-zhang","given":"Lijun","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"Minimizing a convex function of matrices regularized by the nuclear norm arises in many applications such as collaborative filtering and multi-task learning. In this paper, we study the general setting where the convex function could be non-smooth. When the size of the data matrix, denoted by m x n, is very large, existing optimization methods are inefficient because in each iteration, they need to perform a singular value decomposition (SVD) which takes O(m^2 n) time. To reduce the computation cost, we exploit the dual characterization of the nuclear norm to introduce a convex-concave optimization problem and design a subgradient-based algorithm without performing SVD. In each iteration, the proposed algorithm only computes the largest singular vector, reducing the time complexity from O(m^2 n) to O(mn). To the best of our knowledge, this is the first SVD-free convex optimization approach for nuclear-norm regularized problems that does not rely on the smoothness assumption. Theoretical analysis shows that the proposed algorithm converges at an optimal O(1/\\sqrt{T}) rate where T is the number of iterations. We also extend our algorithm to the stochastic case where only stochastic subgradients of the convex function are available and a special case that contains an additional non-smooth regularizer (e.g., L1 norm regularizer). We conduct experiments on robust low-rank matrix approximation and link prediction to demonstrate the efficiency of our algorithms.","doi":"10.24963/ijcai.2017/436","url":"https://ijcai.org/proceedings/2017/436","pdf":"https://ijcai.org/proceedings/2017/0436.pdf","markdown":"https://ijcai.org/proceedings/2017/436.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/436","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":436,"pages":{"first":3126,"last":3132},"keywords":["Machine Learning: Machine Learning","Constraints and Satisfiability: Solvers and Tools"],"videos":[],"references":[{"position":1,"raw":"[Abernethy et al., 2009] Jacob Abernethy, Francis Bach, Theodoros Evgeniou, and Jean-Philippe Vert. A new ap- proach to collaborative filtering: Operator estimation with spectral regularization. Journal of Machine Learning Re- search, 10:803-826, 2009.","title":"A new approach to collaborative filtering: Operator estimation with spectral regularization","authors":"Jacob Abernethy, Francis Bach, Theodoros Evgeniou, Jean-Philippe Vert","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Argyriou et al., 2008] Andreas Argyriou, Theodoros Evge- niou, and Massimiliano Pontil. Convex multi-task feature learning. Machine Learning, 73(3):243-272, 2008.","title":"Convex multi-task feature learning","authors":"Andreas Argyriou, Theodoros Evgeniou, Massimiliano Pontil","year":"2008","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baccini et al., 1996] Alain Baccini, Ph. Besse, and A. de Falguerolles. A l 1 -norm PCA and a heuristic approach. In Proceedings of the International Conference on Ordinal and Symbolic Data Analysis, pages 359-368, 1996.","title":"A l 1 -norm PCA and a heuristic approach","authors":"Alain Baccini, Ph Besse, A De Falguerolles","year":"1996","venue":"Proceedings of the International Conference on Ordinal and Symbolic Data Analysis","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2010] Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956-1982, 2010.","title":"A singular value thresholding algorithm for matrix completion","authors":"Jian-Feng Cai, Emmanuel J Candès, Zuowei Shen","year":"2010","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Candès and Recht, 2009] Emmanuel J. Candès and Ben- jamin Recht. Exact matrix completion via convex op- timization. Foundations of Computational Mathematics, 9(6):717-772, 2009.","title":"Exact matrix completion via convex optimization","authors":"Emmanuel J Candès, Benjamin Recht","year":"2009","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Croux and Filzmoser, 1998] Christophe Croux and Peter Filzmoser. Robust factorization of a data matrix. In Proceedings in Computational Statistics, pages 245-250, 1998.","title":"Robust factorization of a data matrix","authors":"Christophe Croux, Peter Filzmoser","year":"1998","venue":"Proceedings in Computational Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duchi and Singer, 2009] John Duchi and Yoram Singer. Ef- ficient online and batch learning using forward back- ward splitting. Journal of Machine Learning Research, 10(Dec):2899-2934, 2009.","title":"Efficient online and batch learning using forward backward splitting","authors":"Singer Duchi, John Duchi, Yoram Singer","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dudık et al., 2012] Miroslav Dudık, Zaıd Harchaoui, and Jérôme Malick. Lifted coordinate descent for learning with trace-norm regularization. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics, volume 22, pages 327-336, 2012.","title":"Lifted coordinate descent for learning with trace-norm regularization","authors":"Dudık","year":"2012","venue":"Proceedings of the 15th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hazan, 2008] Elad Hazan. Sparse approximate solutions to semidefinite programs. In Proceedings of the 8th Latin American Conference on Theoretical Informatics, pages 306-316, 2008.","title":"Sparse approximate solutions to semidefinite programs","authors":"Elad Hazan","year":"2008","venue":"Proceedings of the 8th Latin American Conference on Theoretical Informatics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hsieh and Olsen, 2014] Cho-Jui Hsieh and Peder A Olsen. Nuclear norm minimization via active subspace selection. In Proceedings of The 31st International Conference on Machine Learning, pages 575-583, 2014.","title":"Nuclear norm minimization via active subspace selection","authors":"Cho-Jui Hsieh, Peder A Olsen","year":"2014","venue":"Proceedings of The 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaggi et al., 2010] Martin Jaggi, Marek Sulovsk, et al. A simple algorithm for nuclear norm regularized problems. In Proceedings of the 27th International Conference on Machine Learning, pages 471-478, 2010.","title":"A simple algorithm for nuclear norm regularized problems","authors":"Martin Jaggi, Marek Sulovsk","year":"2010","venue":"Proceedings of the 27th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaggi, 2013] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Proceed- ings of the 30th International Conference on Machine Learning, pages 427-435, 2013.","title":"Revisiting frank-wolfe: Projection-free sparse convex optimization","authors":"Martin Jaggi, Jaggi","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ji and Ye, 2009] Shuiwang Ji and Jieping Ye. An acceler- ated gradient method for trace norm minimization. In Pro- ceedings of the 26th Annual International Conference on Machine Learning, pages 457-464, 2009.","title":"Shuiwang Ji and Jieping Ye. An accelerated gradient method for trace norm minimization","authors":"Ji, Ye","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ke and Kanade, 2005] Qifa Ke and Takeo Kanade. Robust l 1 norm factorization in the presence of outliers and miss- ing data by alternative convex programming. In Proceed- ings of the IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition, volume 1, pages 739-746, 2005.","title":"Robust l 1 norm factorization in the presence of outliers and missing data by alternative convex programming","authors":"Kanade Ke, Qifa Ke, Takeo Kanade","year":"2005","venue":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lan, 2012] Guanghui Lan. An optimal method for stochas- tic composite optimization. Mathematical Programming, 133:365-397, 2012.","title":"An optimal method for stochastic composite optimization","authors":"Lan ;, Guanghui Lan","year":"2012","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Qihang Lin, Xi Chen, and Javier Peña. A sparsity preserving stochastic gradient methods for sparse regression. Computational Optimization and Applications, 58(2):455-482, 2014.","title":"A sparsity preserving stochastic gradient methods for sparse regression","authors":"Lin","year":"2014","venue":"Computational Optimization and Applications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mahdavi et al., 2012] Mehrdad Mahdavi, Tianbao Yang, Rong Jin, Shenghuo Zhu, and Jinfeng Yi. Stochastic gradi- ent descent with only one projection. In Advances in Neu- ral Information Processing Systems (NIPS), pages 503- 511, 2012.","title":"Stochastic gradient descent with only one projection","authors":"Mehrdad Mahdavi, Tianbao Yang, Rong Jin, Shenghuo Zhu, Jinfeng Yi","year":"2012","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nemirovski et al., 1982] Arkadi Nemirovski, David Boriso- vich Yudin, and E-R Dawson. Problem complexity and method efficiency in optimization. Wiley-Interscience se- ries in discrete mathematics. Wiley, 1982.","title":"Problem complexity and method efficiency in optimization","authors":"Arkadi Nemirovski, David Borisovich Yudin, E-R Dawson","year":"1982","venue":"Wiley-Interscience series in discrete mathematics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nesterov, 2004] Yurii Nesterov. Introductory lectures on convex optimization: a basic course, volume 87 of Applied optimization. Kluwer Academic Publishers, 2004.","title":"Introductory lectures on convex optimization: a basic course","authors":"Yurii Nesterov","year":"2004","venue":"Applied optimization","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nesterov, 2013] Yu Nesterov. Gradient methods for mini- mizing composite functions. Mathematical Programming, 140(1):125-161, 2013.","title":"Gradient methods for minimizing composite functions","authors":"Yu Nesterov","year":"2013","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pong et al., 2010] Ting Kei Pong, Paul Tseng, Shuiwang Ji, and Jieping Ye. Trace norm regularization: Reformula- tions, algorithms, and multi-task learning. SIAM Journal on Optimization, 20(6):3465-3489, 2010.","title":"Trace norm regularization: Reformulations, algorithms, and multi-task learning","authors":"Ting Kei, Pong, Paul Tseng, Shuiwang Ji, Jieping Ye","year":"2010","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rennie and Srebro, 2005] Jasson DM Rennie and Nathan Srebro. Fast maximum margin matrix factorization for col- laborative prediction. In Proceedings of the 22nd Interna- tional Conference on Machine Learning, pages 713-719, 2005.","title":"Fast maximum margin matrix factorization for collaborative prediction","authors":"Rennie, Srebro ; ] Jasson, D M Rennie, Nathan Srebro","year":"2005","venue":"Proceedings of the 22nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Richard et al., 2012] Emile Richard, Pierre-Andre Savalle, and Nicolas Vayatis. Estimation of simultaneously sparse and low rank matrices. In Proceedings of the 29th Inter- national Conference on Machine Learning, pages 1351- 1358, 2012.","title":"Estimation of simultaneously sparse and low rank matrices","authors":"Richard","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shalev-Shwartz et al., 2011] Shai Shalev-Shwartz, Alon Gonen, and Ohad Shamir. Large-scale convex minimiza- tion with a low-rank constraint. In Proceedings of the 28th International Conference on Machine Learning, pages 329-336, 2011.","title":"Large-scale convex minimization with a low-rank constraint","authors":"Shai Shalev-Shwartz, Alon Gonen, Ohad Shamir","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Signoretto et al., 2013] Marco Signoretto, Volkan Cevher, and JA Suykens. An svd-free approach to a class of struc- tured low rank matrix optimization problems with applica- tion to system identification. In Proceedings of the IEEE Conference on Decision and Control, 2013.","title":"An svd-free approach to a class of structured low rank matrix optimization problems with application to system identification","authors":"Marco Signoretto, Volkan Cevher, J A Suykens","year":"2013","venue":"Proceedings of the IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Srebro et al., 2005] Nathan Srebro, Jason Rennie, and Tommi S. Jaakkola. Maximum-margin matrix factoriza- tion. In Advances in Neural Information Processing Sys- tems 17, pages 1329-1336, 2005.","title":"Maximum-margin matrix factorization","authors":"Nathan Srebro, Jason Rennie, Tommi S Jaakkola","year":"2005","venue":"Advances in Neural Information Processing Systems 17","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Toh and Yun, 2010] Kim-Chuan Toh and Sangwoon Yun. An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of Optimization, 6(615-640):15, 2010.","title":"An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems","authors":"Yun ; ] Kim-Chuan Toh, Sangwoon Toh, Yun","year":"2010","venue":"Pacific Journal of Optimization","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}