{"type":"ScholarlyArticle","title":"Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization","authors":[{"name":"Zebang Shen","slug":"zebang-shen","given":"Zebang","family":"Shen","affiliation":"College of Computer Science and Technology , Zhejiang University , China"},{"name":"Hui Qian","slug":"hui-qian","given":"Hui","family":"Qian","affiliation":"College of Computer Science and Technology , Zhejiang University , China"},{"name":"Tongzhou Mu","slug":"tongzhou-mu","given":"Tongzhou","family":"Mu","affiliation":"College of Computer Science and Technology , Zhejiang University , China"},{"name":"Chao Zhang","slug":"chao-zhang","given":"Chao","family":"Zhang","affiliation":"College of Computer Science and Technology , Zhejiang University , China"}],"abstract":"Nowadays, algorithms with fast convergence, small memory footprints, and low per-iteration complexity are particularly favorable for artificial intelligence applications. In this paper, we propose a doubly stochastic algorithm with a novel accelerating multi-momentum technique to solve large scale empirical risk minimization problem for learning tasks. While enjoying a provably superior convergence rate, in each iteration, such algorithm only accesses a mini batch of samples and meanwhile updates a small block of variable coordinates, which substantially reduces the amount of memory reference when both the massive sample size and ultra-high dimensionality are involved. Specifically, to obtain an ε-accurate solution, our algorithm requires only O(log(1/ε)/sqrt(ε)) overall computation for the general convex case and O((n+sqrt{nκ})log(1/ε)) for the strongly convex case. Empirical studies on huge scale datasets are conducted to illustrate the efficiency of our method in practice.","doi":"10.24963/ijcai.2017/378","url":"https://ijcai.org/proceedings/2017/378","pdf":"https://ijcai.org/proceedings/2017/0378.pdf","markdown":"https://ijcai.org/proceedings/2017/378.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/378","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":378,"pages":{"first":2715,"last":2721},"keywords":["Machine Learning: Classification","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Allen-Zhu and Orecchia, 2014] Zeyuan Allen-Zhu and Lorenzo Orecchia. Linear coupling: An ultimate unifi- cation of gradient and mirror descent. arXiv preprint arXiv:1407.1537, 2014.","title":"Linear coupling: An ultimate unification of gradient and mirror descent","authors":"Allen-Zhu Orecchia ; Zeyuan Allen-Zhu, Lorenzo Orecchia","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen-Zhu, 2016] Zeyuan Allen-Zhu. Katyusha: Accel- erated variance reduction for faster sgd. arXiv preprint arXiv:1603.05953, 2016.","title":"Katyusha: Accelerated variance reduction for faster sgd","authors":"Allen-Zhu ; Zeyuan Allen-Zhu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM journal on imaging sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih- Jen Lin. LIBSVM: A library for support vector ma- chines. ACM Transactions on Intelligent Systems and Technology, 2011.","title":"LIBSVM: A library for support vector machines","authors":"Chang, Lin ; Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Defazio et al., 2014] Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gra- dient method with support for non-strongly convex composite objectives. In NIPS, 2014.","title":"Saga: A fast incremental gradient method with support for non-strongly convex composite objectives","authors":"Aaron Defazio, Francis Bach, Simon Lacoste-Julien","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fercoq and Richtárik, 2015] Olivier Fercoq and Peter Richtárik. Accelerated, parallel, and proximal coor- dinate descent. SIAM Journal on Optimization, 2015.","title":"Olivier Fercoq and Peter Richtárik. Accelerated, parallel, and proximal coordinate descent","authors":"Richtárik Fercoq","year":"2015","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Friedman et al., 2001] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning. 2001. [Frostig et al., 2015]","title":"The elements of statistical learning","authors":"Jerome Friedman, Trevor Hastie, Robert Tibshirani","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Roy Frostig, Rong Ge, Sham M Kakade, and Aaron Sidford. Un-regularizing: approx- imate proximal point and faster stochastic algorithms for empirical risk minimization. In ICML, 2015.","title":"Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization","authors":"Roy Frostig, Rong Ge, Aaron Sham M Kakade, Sidford","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2009] Chonghai Hu, Weike Pan, and James T Kwok. Accelerated gradient methods for stochastic optimization and online learning. In Advances in Neu- ral Information Processing Systems, 2009.","title":"Accelerated gradient methods for stochastic optimization and online learning","authors":"Hu","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In NIPS, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lan, 2012] Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Pro- gramming, 2012.","title":"An optimal method for stochastic composite optimization","authors":"Lan ;, Guanghui Lan","year":"2012","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee and Sidford, 2013] Yin Tat Lee and Aaron Sidford. Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems. In FOCS, 2013.","title":"Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems","authors":"Sidford ; ] Lee, Tat Yin, Aaron Lee, Sidford","year":"2013","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2015a] Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. In NIPS, 2015.","title":"A universal catalyst for first-order optimization","authors":"Lin","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2015b] Qihang Lin, Zhaosong Lu, and Lin Xiao. An accelerated randomized proximal coordinate gradient method and its application to regularized em- pirical risk minimization. SIAM Journal on Optimiza- tion, 2015.","title":"An accelerated randomized proximal coordinate gradient method and its application to regularized empirical risk minimization","authors":"Lin","year":"2015","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nesterov, 1983] Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, 1983.","title":"A method of solving a convex programming problem with convergence rate o (1/k2)","authors":"Yurii Nesterov","year":"1983","venue":"Soviet Mathematics Doklady","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nesterov, 1998] Yu Nesterov. Introductory lectures on convex programming volume i: Basic course. 1998.","title":"Introductory lectures on convex programming","authors":"Yu Nesterov","year":"1998","venue":"Basic course","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nesterov, 2012] Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 2012.","title":"Efficiency of coordinate descent methods on huge-scale optimization problems","authors":"Yu Nesterov","year":"2012","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nitanda, 2014] Atsushi Nitanda. Stochastic proximal gradient descent with acceleration techniques. In NIPS, 2014.","title":"Stochastic proximal gradient descent with acceleration techniques","authors":"Atsushi Nitanda","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Polyak, 1964] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 1964.","title":"Some methods of speeding up the convergence of iteration methods","authors":"T Boris, Polyak","year":"1964","venue":"USSR Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Richtárik and Takáč, ] Peter Richtárik and Martin Takáč. Iteration complexity of randomized block- coordinate descent methods for minimizing a compos- ite function. Mathematical Programming.","title":"Iteration complexity of randomized blockcoordinate descent methods for minimizing a composite function","authors":"Takáč Richtárik, ] Peter Richtárik, Martin Takáč","year":null,"venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schmidt et al., 2013] Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. arXiv preprint arXiv:1309.2388, 2013.","title":"Minimizing finite sums with the stochastic average gradient","authors":"Mark Schmidt, Nicolas Le Roux, Francis Bach","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shalev-Shwartz and Zhang, 2014] Shai Shalev-Shwartz and Tong Zhang. Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. In ICML, 2014.","title":"Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization","authors":"Shalev-Shwartz Zhang, ; ] Shai Shalev-Shwartz, Tong Zhang","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simon et al., 2013] Noah Simon, Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A sparse-group lasso. Journal of Computational and Graphical Statis- tics, 2013.","title":"A sparse-group lasso","authors":"Simon","year":"2013","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrink- age and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society. Series B (Methodological","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Woodworth and Srebro, 2016] Blake E Woodworth and Nati Srebro. Tight complexity bounds for optimizing composite objectives. In NIPS, 2016.","title":"Tight complexity bounds for optimizing composite objectives","authors":"E Blake, Nati Woodworth, Srebro","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wright, 2015] Stephen J Wright. Coordinate descent algorithms. Mathematical Programming, 2015.","title":"Coordinate descent algorithms","authors":"Wright Stephen","year":"2015","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Gu, 2016] Aston Zhang and Quanquan Gu. Accelerated stochastic block coordinate descent with optimal sampling. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.","title":"Accelerated stochastic block coordinate descent with optimal sampling","authors":"Gu Zhang, Aston Zhang, Quanquan Gu","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Xiao, 2015] Yuchen Zhang and Lin Xiao. Stochastic primal-dual coordinate method for regular- ized empirical risk minimization. In ICML, 2015.","title":"Stochastic primal-dual coordinate method for regularized empirical risk minimization","authors":"Yuchen Zhang, Lin Xiao","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2014] Tuo Zhao, Mo Yu, Yiming Wang, Ra- man Arora, and Han Liu. Accelerated mini-batch ran- domized block coordinate descent method. In NIPS, 2014.","title":"Accelerated mini-batch randomized block coordinate descent method","authors":"Tuo Zhao, Mo Yu, Yiming Wang, Raman Arora, Han Liu","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}