{"type":"ScholarlyArticle","title":"Adaptive Variance Reducing for Stochastic Gradient Descent","authors":[{"name":"Zebang Shen","slug":"zebang-shen","given":"Zebang","family":"Shen","affiliation":"Zhejiang University , China"},{"name":"Hui Qian","slug":"hui-qian","given":"Hui","family":"Qian","affiliation":"Zhejiang University , China"},{"name":"Tengfei Zhou","slug":"tengfei-zhou","given":"Tengfei","family":"Zhou","affiliation":"Zhejiang University , China"},{"name":"Tongzhou Mu","slug":"tongzhou-mu","given":"Tongzhou","family":"Mu","affiliation":"Zhejiang University , China"}],"abstract":"Variance Reducing (VR) stochastic methods are fast-converging alternatives to the classical Stochastic Gradient Descent (SGD) for solving large-scale regularized finite sum problems, especially when a highly accurate solution is required. One critical step in VR is the function sampling. State-of-the-art VR algorithms such as SVRG and SAGA, employ either Uniform Probability (UP) or Importance Probability (IP), which is deficient in reducing the variance and hence leads to suboptimal convergence rate. In this paper, we propose a novel sampling scheme that explicitly computes some Adaptive Probability (AP) at each iteration. Analysis shows that, equipped with AP, both SVRG and SAGA yield provably better convergence rate than the ones with UP or IP, which is confirmed in experiments. Additionally, to cut down the per iteration computation load, an efficient variant is proposed by utilizing AP periodically, whose performance is empirically validated.","doi":null,"url":"https://ijcai.org/proceedings/2016/284","pdf":"https://ijcai.org/proceedings/2016/0284.pdf","markdown":"https://ijcai.org/proceedings/2016/284.md","bibtex":"https://ijcai.org/proceedings/2016/bibtex/284","volume":{"slug":"2016","title":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2016","url":"https://ijcai.org/proceedings/2016/","year":2016,"location":"New York, NY, USA","dates":"9–15 July 2016","editors":"Subbarao Kambhampati","isbn":"978-1-57735-770-4; 978-1-57735-771-1","publisher":"AAAI Press / International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":284,"pages":{"first":1990,"last":1995},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for lin- ear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM journal on imaging sciences","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bottou, 2010] Léon Bottou. Large-scale machine learn- ing with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pages 177-186. Springer, 2010.","title":"Large-scale machine learning with stochastic gradient descent","authors":"Léon Bottou, Bottou","year":"2010","venue":"Proceedings of COMPSTAT'2010","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011.","title":"LIBSVM: A library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Rockafellar, 1997] George HG Chen and RT Rockafellar. Convergence rates in forward-backward splitting. SIAM Journal on Optimization, 7(2):421-444, 1997.","title":"Convergence rates in forward-backward splitting","authors":"Chen, Rockafellar ; ] George, H G Chen, R T Rockafellar","year":"1997","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Csiba et al., 2015] Dominik Csiba, Zheng Qu, and Peter Richtárik. Stochastic dual coordinate ascent with adaptive probabilities. arXiv preprint arXiv:1502.08053, 2015.","title":"Stochastic dual coordinate ascent with adaptive probabilities","authors":"Dominik Csiba, Zheng Qu, Peter Richtárik","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Defazio et al., 2014] Aaron Defazio, Francis Bach, and Si- mon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems, pages 1646-1654, 2014.","title":"Saga: A fast incremental gradient method with support for non-strongly convex composite objectives","authors":"Aaron Defazio, Francis Bach, Simon Lacoste-Julien","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learn- ing and stochastic optimization. The Journal of Machine Learning Research, 12:2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hastie et al., 2005] Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2):83-85, 2005.","title":"The elements of statistical learning: data mining, inference and prediction","authors":"Trevor Hastie, Robert Tibshirani, Jerome Friedman, James Franklin","year":"2005","venue":"The Mathematical Intelligencer","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2009] Chonghai Hu, Weike Pan, and James T Kwok. Accelerated gradient methods for stochastic op- timization and online learning. In Advances in Neural In- formation Processing Systems, pages 781-789, 2009.","title":"Accelerated gradient methods for stochastic optimization and online learning","authors":"Hu","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konečnỳ and Richtárik, 2013] Jakub Konečnỳ and Peter Richtárik. Semi-stochastic gradient descent methods. arXiv preprint arXiv:1312.1666, 2013.","title":"Jakub Konečnỳ and Peter Richtárik. Semi-stochastic gradient descent methods","authors":"Richtárik Konečnỳ","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Konecny et al., 2015] Jakub Konecny, Jie Liu, Peter Richtárik, and Martin Takác. Mini-batch semi-stochastic gradient descent in the proximal setting. 2015.","title":"Mini-batch semi-stochastic gradient descent in the proximal setting","authors":"Jakub Konecny, Jie Liu, Peter Richtárik, Martin Takác","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lewis et al., 2004] David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361-397, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"Yiming David D Lewis, Tony G Yang, Fan Rose, Li","year":"2004","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahdavi et al., 2013] Mehrdad Mahdavi, Lijun Zhang, and Rong Jin. Mixed optimization for smooth functions. In Ad- vances in Neural Information Processing Systems, pages 674-682, 2013.","title":"Mixed optimization for smooth functions","authors":"Mehrdad Mahdavi, Lijun Zhang, Rong Jin","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nesterov, 1998] Yu Nesterov. Introductory lectures on con- vex programming volume i: Basic course. 1998.","title":"Basic course","authors":"Yu Nesterov, Nesterov","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nesterov, 2012] Yu Nesterov. Efficiency of coordinate de- scent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-362, 2012.","title":"Efficiency of coordinate descent methods on huge-scale optimization problems","authors":"Yu Nesterov","year":"2012","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nesterov, 2013] Yu Nesterov. Gradient methods for mini- mizing composite functions. Mathematical Programming, 140(1):125-161, 2013.","title":"Gradient methods for minimizing composite functions","authors":"Yu Nesterov","year":"2013","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schmidt et al., 2013] Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. arXiv preprint arXiv:1309.2388, 2013.","title":"Minimizing finite sums with the stochastic average gradient","authors":"Mark Schmidt, Nicolas Le Roux, Francis Bach","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schmidt et al., 2015] Mark Schmidt, Reza Babanezhad, Mohamed Osama Ahmed, Aaron Defazio, Ann Clifton, and Anoop Sarkar. Non-uniform stochastic average gra- dient method for training conditional random fields. arXiv preprint arXiv:1504.04406, 2015.","title":"Non-uniform stochastic average gradient method for training conditional random fields","authors":"Mark Schmidt, Reza Babanezhad, Mohamed Osama Ahmed, Aaron Defazio, Ann Clifton, Anoop Sarkar","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shalev-Shwartz and Zhang, 2013a] Shai Shalev-Shwartz and Tong Zhang. Accelerated mini-batch stochastic dual coordinate ascent. In Advances in Neural Information Processing Systems, pages 378-385, 2013.","title":"Accelerated mini-batch stochastic dual coordinate ascent","authors":"Shalev-Shwartz Zhang ; A] Shai Shalev-Shwartz, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shalev-Shwartz and Zhang, 2013b] Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss. The Journal of Machine Learning Research, 14(1):567-599, 2013.","title":"Stochastic dual coordinate ascent methods for regularized loss","authors":"Shalev-Shwartz Zhang, Shai Shalev-Shwartz, Tong Zhang","year":"2013","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shalev-Shwartz and Zhang, 2016] Shai Shalev-Shwartz and Tong Zhang. Accelerated proximal stochastic dual coordi- nate ascent for regularized loss minimization. Mathemati- cal Programming, 155(1-2):105-145, 2016.","title":"Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization","authors":"Shalev-Shwartz Zhang, ; ] Shai Shalev-Shwartz, Tong Zhang","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vitter, 1987] Jeffrey Scott Vitter. Design and analysis of dynamic huffman codes. Journal of the ACM (JACM), 34(4):825-845, 1987.","title":"Design and analysis of dynamic huffman codes","authors":"Jeffrey Scott, Vitter","year":"1987","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiao and Zhang, 2014] Lin Xiao and Tong Zhang. A prox- imal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization, 24(4):2057- 2075, 2014.","title":"A proximal stochastic gradient method with progressive variance reduction","authors":"Lin Xiao, Tong Zhang","year":"2014","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao and Zhang, 2014] Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling. arXiv preprint arXiv:1401.2753, 2014.","title":"Stochastic optimization with importance sampling","authors":"Peilin Zhao, Tong Zhang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}