{"type":"ScholarlyArticle","title":"Stochastic Second-Order Method for Large-Scale Nonconvex Sparse Learning Models","authors":[{"name":"Hongchang Gao","slug":"hongchang-gao","given":"Hongchang","family":"Gao","affiliation":"Department of Electrical and Computer Engineering , University of Pittsburgh , USA"},{"name":"Heng Huang","slug":"heng-huang","given":"Heng","family":"Huang","affiliation":"Department of Electrical and Computer Engineering , University of Pittsburgh , USA"}],"abstract":"Sparse learning models have shown promising performance in the high dimensional machine learning applications. The main challenge of sparse learning models is how to optimize it efficiently. Most existing methods solve this problem by relaxing it as a convex problem, incurring large estimation bias. Thus, the sparse learning model with nonconvex constraint has attracted much attention due to its better performance. But it is difficult to optimize due to the non-convexity.\r\nIn this paper, we propose a linearly convergent stochastic second-order method to optimize this nonconvex problem for large-scale datasets. The proposed method incorporates second-order information to improve the convergence speed. Theoretical analysis shows that our proposed method enjoys linear convergence rate and guarantees to converge to the underlying true model parameter. Experimental results have verified the efficiency and correctness of our proposed method.","doi":"10.24963/ijcai.2018/294","url":"https://ijcai.org/proceedings/2018/294","pdf":"https://ijcai.org/proceedings/2018/0294.pdf","markdown":"https://ijcai.org/proceedings/2018/294.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/294","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":294,"pages":{"first":2128,"last":2134},"keywords":["Machine Learning Applications: Big data ; Scalability","Machine Learning: Feature Selection ; Learning Sparse Models"],"videos":[],"references":[{"position":1,"raw":"[Banerjee et al., 2008] Onureena Banerjee, Laurent El Ghaoui, and Alexandre dAspremont. Model selection through sparse maximum likelihood estimation for mul- tivariate gaussian or binary data. Journal of Machine learning research, 9(Mar):485-516, 2008.","title":"Onureena Banerjee, Laurent El Ghaoui, and Alexandre dAspremont. Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data","authors":"Banerjee","year":"2008","venue":"Journal of Machine learning research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blumensath and Davies, 2009] Thomas Blumensath and Mike E Davies. Iterative hard thresholding for compressed sensing. Applied and computational harmonic analysis, 27(3):265-274, 2009.","title":"Iterative hard thresholding for compressed sensing","authors":"Thomas Blumensath, Mike E Davies","year":"2009","venue":"Applied and computational harmonic analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Byrd et al., 2016] Richard H Byrd, Samantha L Hansen, Jorge Nocedal, and Yoram Singer. A stochastic quasi- newton method for large-scale optimization. SIAM Jour- nal on Optimization, 26(2):1008-1031, 2016.","title":"A stochastic quasinewton method for large-scale optimization","authors":"Samantha L Richard H Byrd, Jorge Hansen, Yoram Nocedal, Singer","year":"2016","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Gu, 2016] Jinghui Chen and Quanquan Gu. Ac- celerated stochastic block coordinate gradient descent for sparsity constrained nonconvex optimization. In Confer- ence on Uncertainty in Artificial Intelligence, 2016.","title":"Accelerated stochastic block coordinate gradient descent for sparsity constrained nonconvex optimization","authors":"Jinghui Chen, Quanquan Gu","year":"2016","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Friedman et al., 2008] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estima- tion with the graphical lasso. Biostatistics, 9(3):432-441, 2008.","title":"Sparse inverse covariance estimation with the graphical lasso","authors":"Jerome Friedman, Trevor Hastie, Robert Tibshirani","year":"2008","venue":"Biostatistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2015] Hongchang Gao, Lin Yan, Weidong Cai, and Heng Huang. Anatomical annotations for drosophila gene expression patterns via multi-dimensional visual de- scriptors integration: Multi-dimensional feature learning. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 339-348. ACM, 2015.","title":"Anatomical annotations for drosophila gene expression patterns via multi-dimensional visual descriptors integration: Multi-dimensional feature learning","authors":"Hongchang Gao, Lin Yan, Weidong Cai, Heng Huang","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2017] Hongchang Gao, Feiping Nie, and Heng Huang. Local centroids structured non-negative matrix factorization. In Thirty-First AAAI Conference on Artifi- cial Intelligence, 2017.","title":"Local centroids structured non-negative matrix factorization","authors":"Gao","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gower et al., 2016] Robert Gower, Donald Goldfarb, and Peter Richtárik. Stochastic block bfgs: squeezing more curvature out of data. In International Conference on Ma- chine Learning, pages 1869-1878, 2016.","title":"Stochastic block bfgs: squeezing more curvature out of data","authors":"Robert Gower, Donald Goldfarb, Peter Richtárik","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2006] Su-In Lee, Honglak Lee, Pieter Abbeel, and Andrew Y Ng. Efficient l˜1 regularized logistic re- gression. In AAAI, 2006.","title":"Efficient l˜1 regularized logistic regression","authors":"Lee","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Xingguo Li, Raman Arora, Han Liu, Jarvis Haupt, and Tuo Zhao. Nonconvex sparse learning via stochastic optimization with progressive variance reduc- tion. arXiv preprint arXiv:1605.02711, 2016.","title":"Nonconvex sparse learning via stochastic optimization with progressive variance reduction","authors":"Xingguo Li, Raman Arora, Han Liu, Jarvis Haupt, Tuo Zhao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mallat and Zhang, 1993] Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dic- tionaries. IEEE Transactions on signal processing, 41(12):3397-3415, 1993.","title":"Matching pursuits with time-frequency dictionaries","authors":"G Stéphane, Zhifeng Mallat, Zhang","year":"1993","venue":"IEEE Transactions on signal processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moritz et al., 2016] Philipp Moritz, Robert Nishihara, and Michael Jordan. A linearly-convergent stochastic l-bfgs algorithm. In Artificial Intelligence and Statistics, pages 249-258, 2016.","title":"A linearly-convergent stochastic l-bfgs algorithm","authors":"Philipp Moritz, Robert Nishihara, Michael Jordan","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Needell and Tropp, 2009] Deanna Needell and Joel A Tropp. Cosamp: Iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Har- monic Analysis, 26(3):301-321, 2009.","title":"Cosamp: Iterative signal recovery from incomplete and inaccurate samples","authors":"Deanna Needell, Joel A Tropp","year":"2009","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nguyen et al., 2014] Nam Nguyen, Deanna Needell, and Tina Woolf. Linear convergence of stochastic iterative greedy algorithms with sparse constraints. arXiv preprint arXiv:1407.0088, 2014.","title":"Linear convergence of stochastic iterative greedy algorithms with sparse constraints","authors":"Nguyen","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nocedal and Wright, 2006] Jorge Nocedal and Stephen J Wright. Numerical optimization (2nd edition), 2006.","title":"Numerical optimization","authors":"Jorge Nocedal, Stephen J Wright","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267-288, 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society. Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tropp and Gilbert, 2007] Joel A Tropp and Anna C Gilbert. Signal recovery from random measurements via orthogo- nal matching pursuit. IEEE Transactions on information theory, 53(12):4655-4666, 2007.","title":"Signal recovery from random measurements via orthogonal matching pursuit","authors":"Gilbert ; Tropp, Joel A Tropp, Anna C Gilbert","year":"2007","venue":"IEEE Transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van de Geer, 2008] Sara A Van de Geer. High-dimensional generalized linear models and the lasso. The Annals of Statistics, pages 614-645, 2008.","title":"High-dimensional generalized linear models and the lasso","authors":"A Sara, Van De Geer","year":"2008","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yuan and Lin, 2006] Ming Yuan and Yi Lin. Model selec- tion and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statisti- cal Methodology), 68(1):49-67, 2006.","title":"Model selection and estimation in regression with grouped variables","authors":"Lin ; Yuan, Ming Yuan, Yi Lin","year":"2006","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yuan and Liu, 2014] Xiao-Tong Yuan and Qingshan Liu. Newton greedy pursuit: A quadratic approximation method for sparsity-constrained optimization. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 4122-4129, 2014.","title":"Newton greedy pursuit: A quadratic approximation method for sparsity-constrained optimization","authors":"Xiao-Tong Yuan, Qingshan Liu","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yuan et al., 2014] Xiaotong Yuan, Ping Li, and Tong Zhang. Gradient hard thresholding pursuit for sparsity- constrained optimization. In Proceedings of the 31st In- ternational Conference on Machine Learning (ICML-14), pages 127-135, 2014.","title":"Gradient hard thresholding pursuit for sparsityconstrained optimization","authors":"Xiaotong Yuan, Ping Li, Tong Zhang","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning (ICML-14)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang, 2011] Tong Zhang. Adaptive forward-backward greedy algorithm for learning sparse representations. IEEE transactions on information theory, 57(7):4689-4708, 2011.","title":"Adaptive forward-backward greedy algorithm for learning sparse representations","authors":"Tong Zhang","year":"2011","venue":"IEEE transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2017] Renbo Zhao, William B Haskell, and Vincent YF Tan. Stochastic l-bfgs revisited: Improved convergence rates and practical acceleration strategies. arXiv preprint arXiv:1704.00116, 2017.","title":"Stochastic l-bfgs revisited: Improved convergence rates and practical acceleration strategies","authors":"Zhao","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}