{"type":"ScholarlyArticle","title":"Sketched Iterative Algorithms for Structured Generalized Linear Models","authors":[{"name":"Qilong Gu","slug":"qilong-gu","given":"Qilong","family":"Gu","affiliation":"Department of Computer Science & Engineering , University of Minnesota , Twin-Cities"},{"name":"Arindam Banerjee","slug":"arindam-banerjee","given":"Arindam","family":"Banerjee","affiliation":"Department of Computer Science & Engineering , University of Minnesota , Twin-Cities"}],"abstract":"Recent years have seen advances in optimizing large scale statistical estimation problems. In statistical learning settings iterative optimization algorithms have been shown to enjoy geometric convergence. While powerful, such results only hold for the original dataset, and may face computational challenges when the sample size is large. In this paper, we study sketched iterative algorithms, in particular sketched-PGD (projected gradient descent) and sketched-SVRG (stochastic variance reduced gradient) for structured generalized linear model, and illustrate that these methods continue to have geometric convergence to the statistical error under suitable assumptions. Moreover, the sketching dimension is allowed to be even smaller than the ambient dimension, thus can lead to significant speed-ups. The sketched iterative algorithms introduced provide an additional dimension to study the trade-offs between statistical accuracy and time.","doi":"10.24963/ijcai.2019/332","url":"https://ijcai.org/proceedings/2019/332","pdf":"https://ijcai.org/proceedings/2019/0332.pdf","markdown":"https://ijcai.org/proceedings/2019/332.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/332","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":332,"pages":{"first":2392,"last":2398},"keywords":["Machine Learning: Dimensionality Reduction and Manifold Learning","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2012] Alekh Agarwal, Sahand Negahban, and Martin J. Wainwright. Fast global convergence of gradient methods for high-dimensional statistical recovery. The Annals of Statistics, 40(5):2452-2482, 10 2012.","title":"Fast global convergence of gradient methods for high-dimensional statistical recovery","authors":"Alekh Agarwal, Sahand Negahban, Martin J Wainwright","year":"2012","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ailon and Liberty, 2008] Nir Ailon and Edo Liberty. Fast dimension reduction using Rademacher series on dual BCH codes. Discrete & Computational Geometry, 42(4):615, Sep 2008.","title":"Nir Ailon and Edo Liberty. Fast dimension reduction using Rademacher series on dual BCH codes","authors":"Liberty Ailon","year":"2008","venue":"Discrete & Computational Geometry","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Argyriou et al., 2012] Andreas Argyriou, Rina Foygel, and Nathan Srebro. Sparse Prediction with the k-Support Norm. In NIPS, pages 1457-1465, 2012.","title":"Sparse Prediction with the k-Support Norm","authors":"Andreas Argyriou, Rina Foygel, Nathan Srebro","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banerjee et al., 2014] Arindam Banerjee, Sheng Chen, Farideh Fazayeli, and Vidyashankar Sivakumar. Estimation with Norm Regularization. In NIPS, pages 1556-1564, 2014.","title":"Estimation with Norm Regularization","authors":"Arindam Banerjee, Sheng Chen, Farideh Fazayeli, Vidyashankar Sivakumar","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bertsekas, 2010] D. P. Bertsekas. Nonlinear Programming. Athena Scientific, 2010.","title":"Nonlinear Programming","authors":"D P Bertsekas","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chandrasekaran et al., 2012] Venkat Chandrasekaran, Ben- jamin Recht, Pablo A. Parrilo, and Alan S. Willsky. The Convex Geometry of Linear Inverse Problems. Foundations of Computational Mathematics, 12(6):805-849, 2012.","title":"The Convex Geometry of Linear Inverse Problems","authors":"Venkat Chandrasekaran, Benjamin Recht, Pablo A Parrilo, Alan S Willsky","year":"2012","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Defazio et al., 2014] Aaron Defazio, Francis Bach, and Si- mon Lacoste-julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In NIPS, pages 1646-1654, 2014.","title":"Saga: A fast incremental gradient method with support for non-strongly convex composite objectives","authors":"Aaron Defazio, Francis Bach, Simon Lacoste-Julien","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Drineas et al., 2006] P. Drineas, R. Kannan, and M. Ma- honey. Fast monte carlo algorithms for matrices i: Approxi- mating matrix multiplication. SIAM Journal on Computing, 36(1):132-157, 2006.","title":"Fast monte carlo algorithms for matrices i: Approximating matrix multiplication","authors":"P Drineas, R Kannan, M Mahoney","year":"2006","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In NIPS, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2016] Xingguo Li, Tuo Zhao, Raman Arora, Han Liu, and Jarvis Haupt. Stochastic variance reduced opti- mization for nonconvex sparse learning. In ICML, pages 917-925, 2016.","title":"Stochastic variance reduced optimization for nonconvex sparse learning","authors":"Xingguo Li, Tuo Zhao, Raman Arora, Han Liu, Jarvis Haupt","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mahoney, 2011] Michael W. Mahoney. Randomized algo- rithms for matrices and data. Foundations and Trends® in Machine Learning, 3(2):123-224, 2011.","title":"Randomized algorithms for matrices and data","authors":"W Michael, Mahoney","year":"2011","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Negahban et al., 2012] Sahand N. Negahban, Pradeep Ravikumar, Martin J. Wainwright, and Bin Yu. A Uni- fied Framework for High-Dimensional Analysis of M - Estimators with Decomposable Regularizers. Statistical Science, 27(4):538-557, 2012.","title":"A Unified Framework for High-Dimensional Analysis of M -Estimators with Decomposable Regularizers","authors":"N Sahand, Pradeep Negahban, Martin J Ravikumar, Bin Wainwright, Yu","year":"2012","venue":"Statistical Science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oymak et al., 2018] Samet Oymak, Benjamin Recht, and Mahdi Soltanolkotabi. Sharp time-data tradeoffs for lin- ear inverse problems. IEEE Transactions on Information Theory, pages 4129-4158, 2018.","title":"Sharp time-data tradeoffs for linear inverse problems","authors":"Samet Oymak, Benjamin Recht, Mahdi Soltanolkotabi","year":"2018","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pilanci and Wainwright, 2015] Mert Pilanci and Martin J. Wainwright. Randomized sketches of convex programs with sharp guarantees. IEEE Transactions on Information Theory, pages 5096-5115, 2015.","title":"Randomized sketches of convex programs with sharp guarantees","authors":"Mert Pilanci, Martin J Wainwright","year":"2015","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pilanci and Wainwright, 2016] Mert Pilanci and Martin J. Wainwright. Iterative Hessian sketch: Fast and accurate so- lution approximation for constrained least-squares. Journal of Machine Learning Research (JMLR), 17(1):1842-1879, January 2016.","title":"Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares","authors":"Wainwright ; Pilanci, Mert Pilanci, Martin J Wainwright","year":"2016","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pilanci and Wainwright, 2017] Mert Pilanci and Martin J. Wainwright. Newton sketch: A linear-time optimization algorithm with linear-quadratic convergence. SIAM Journal of Optimization, 27(1):205-245, 2017.","title":"Newton sketch: A linear-time optimization algorithm with linear-quadratic convergence","authors":"Mert Pilanci, Martin J Wainwright","year":"2017","venue":"SIAM Journal of Optimization","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qu et al., 2017] Chao Qu, Yan Li, and Huan Xu. Lin- ear Convergence of SVRG in Statistical Estimation. arXiv:1611.01957, 2017.","title":"Linear Convergence of SVRG in Statistical Estimation","authors":"Chao Qu, Yan Li, Huan Xu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Raskutti et al., 2010] Garvesh Raskutti, Martin J. Wain- wright, and Bin Yu. Restricted Eigenvalue Properties for Correlated Gaussian Designs. Journal of Machine Learning Research (JMLR), 11:2241-2259, 2010.","title":"Restricted Eigenvalue Properties for Correlated Gaussian Designs","authors":"Raskutti","year":"2010","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Recht et al., 2010] Benjamin Recht, Maryam Fazel, and Pablo A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):471-501, 2010.","title":"Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization","authors":"Benjamin Recht, Maryam Fazel, Pablo A Parrilo","year":"2010","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400-407, 09 1951.","title":"Herbert Robbins and Sutton Monro. A stochastic approximation method","authors":"Monro Robbins","year":"1951","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schmidt et al., 2017] Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162(1):83- 112, Mar 2017.","title":"Minimizing finite sums with the stochastic average gradient","authors":"Mark Schmidt, Nicolas Le Roux, Francis Bach","year":"2017","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2017] Junqi Tang, Mohammad Golbabaee, and Mike E. Davies. Gradient projection iterative sketch for large-scale constrained least-squares. In ICML, pages 3377- 3386, 2017.","title":"Gradient projection iterative sketch for large-scale constrained least-squares","authors":"Tang","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society, 58(1):267-288, 1996.","title":"Regression Shrinkage and Selection via the Lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vershynin, 2012] Roman Vershynin. Introduction to the non- asymptotic analysis of random matrices. In Compressed Sensing, pages 210-268. Cambridge University Press, Cam- bridge, nov 2012.","title":"Roman Vershynin. Introduction to the nonasymptotic analysis of random matrices","authors":"Vershynin","year":"2012","venue":"Compressed Sensing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vershynin, 2015] Roman Vershynin. Estimation in High Di- mensions: A Geometric Perspective, pages 3-66. Springer International Publishing, Cham, 2015.","title":"Roman Vershynin. Estimation in High Dimensions: A Geometric Perspective","authors":"Vershynin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2017] Jialei Wang, Jason Lee, Mehrdad Mah- davi, Mladen Kolar, and Nati Srebro. Sketching Meets Random Projection in the Dual: A Provable Recovery Al- gorithm for Big and High-dimensional Data. In AISTATS, pages 1150-1158, 2017.","title":"Sketching Meets Random Projection in the Dual: A Provable Recovery Algorithm for Big and High-dimensional Data","authors":"Wang","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Woodruff, 2014] David P. Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1-2):1-157, 2014.","title":"Sketching as a tool for numerical linear algebra","authors":"P David, Woodruff","year":"2014","venue":"Foundations and Trends® in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2013] Lijun Zhang, Mehrdad Mahdavi, Rong Jin, Tianbao Yang, and Shenghuo Zhu. Recovering the Optimal Solution by Dual Random Projection. In COLT, pages 135-157. 2013.","title":"Recovering the Optimal Solution by Dual Random Projection","authors":"Lijun Zhang, Mehrdad Mahdavi, Rong Jin, Tianbao Yang, Shenghuo Zhu","year":"2013","venue":"COLT","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}