{"type":"ScholarlyArticle","title":"Accelerating Stratified Sampling SGD by Reconstructing Strata","authors":[{"name":"Weijie Liu","slug":"weijie-liu","given":"Weijie","family":"Liu","affiliation":"Qiushi Academy for Advanced Studies , Zhejiang University , Hangzhou , Zhejiang , China; College of Computer Science and Technology , Zhejiang University , Hangzhou , Zhejiang , China"},{"name":"Hui Qian","slug":"hui-qian","given":"Hui","family":"Qian","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , Zhejiang , China"},{"name":"Chao Zhang","slug":"chao-zhang","given":"Chao","family":"Zhang","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , Zhejiang , China"},{"name":"Zebang Shen","slug":"zebang-shen","given":"Zebang","family":"Shen","affiliation":"University of Pennsylvania , Philadelphia , Pennsylvania"},{"name":"Jiahao Xie","slug":"jiahao-xie","given":"Jiahao","family":"Xie","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , Zhejiang , China"},{"name":"Nenggan Zheng","slug":"nenggan-zheng","given":"Nenggan","family":"Zheng","affiliation":"Qiushi Academy for Advanced Studies , Zhejiang University , Hangzhou , Zhejiang , China"}],"abstract":"In this paper, a novel stratified sampling strategy is designed to accelerate the mini-batch SGD. We derive a new iteration-dependent surrogate which bound the stochastic variance from above. To keep the strata minimizing this surrogate with high probability, a stochastic stratifying algorithm is adopted in an adaptive manner, that is, in each iteration, strata are reconstructed only if an easily verifiable condition is met. Based on this novel sampling strategy, we propose an accelerated mini-batch SGD algorithm named SGD-RS. Our theoretical analysis shows that the convergence rate of SGD-RS is superior to the state-of-the-art. Numerical experiments corroborate our theory and demonstrate that SGD-RS achieves at least 3.48-times speed-ups compared to vanilla minibatch SGD.","doi":"10.24963/ijcai.2020/378","url":"https://ijcai.org/proceedings/2020/378","pdf":"https://ijcai.org/proceedings/2020/0378.pdf","markdown":"https://ijcai.org/proceedings/2020/378.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/378","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":378,"pages":{"first":2725,"last":2731},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Machine Learning: Clustering","Machine Learning: Deep-learning Theory","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24134/file_24134.m3u8","length_sec":287,"provider":"hls"}],"references":[{"position":1,"raw":"[Borsos et al., 2018] Zalán Borsos, Andreas Krause, and Kfir Y Levy. Online variance reduction for stochastic op- timization. arXiv preprint arXiv:1802.04715, 2018.","title":"Online variance reduction for stochastic optimization","authors":"Borsos","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Botev and Ridder, 2014] Zdravko Botev and Ad Ridder. Variance reduction. Wiley StatsRef: Statistics Reference Online, pages 1-6, 2014.","title":"Variance reduction","authors":"Ridder Botev, Zdravko Botev, Ad Ridder","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Finkel et al., 2008] Jenny Rose Finkel, Alex Kleeman, and Christopher D Manning. Efficient, feature-based, condi- tional random field parsing. In Proceedings of ACL-08: HLT, pages 959-967, 2008.","title":"Efficient, feature-based, conditional random field parsing","authors":"Jenny Rose Finkel, Alex Kleeman, Christopher D Manning","year":"2008","venue":"Proceedings of ACL-08: HLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Johnson and Guestrin, 2018] Tyler B Johnson and Carlos Guestrin. Training deep models faster with robust, ap- proximate importance sampling. In Advances in Neural Information Processing Systems, pages 7265-7275, 2018.","title":"Training deep models faster with robust, approximate importance sampling","authors":"B Tyler, Carlos Johnson, Guestrin","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information pro- cessing systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Katharopoulos and Fleuret, 2018] Angelos Katharopoulos and Franc ¸ois Fleuret. Not all samples are created equal: Deep learning with importance sampling. arXiv preprint arXiv:1803.00942, 2018.","title":"Angelos Katharopoulos and Franc ¸ois Fleuret. Not all samples are created equal: Deep learning with importance sampling","authors":"Fleuret Katharopoulos","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kumar and Kannan, 2010] Amit Kumar and Ravindran Kannan. Clustering with spectral norm and the k-means algorithm. In 2010 IEEE 51st Annual Symposium on Foun- dations of Computer Science, pages 299-308. IEEE, 2010.","title":"Clustering with spectral norm and the k-means algorithm","authors":"Amit Kumar, Ravindran Kannan","year":"2010","venue":"IEEE 51st Annual Symposium on Foundations of Computer Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Needell et al., 2016] Deanna Needell, Nathan Srebro, and Rachel Ward. Stochastic gradient descent, weighted sam- pling, and the randomized kaczmarz algorithm. Mathe- matical Programming, 155(1-2):549-573, 2016.","title":"Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm","authors":"Deanna Needell, Nathan Srebro, Rachel Ward","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Salehi et al., 2017] Farnood Salehi, L Elisa Celis, and Patrick Thiran. Stochastic optimization with bandit sam- pling. arXiv preprint arXiv:1708.02544, 2017.","title":"Stochastic optimization with bandit sampling","authors":"Farnood Salehi, Elisa Celis, Patrick Thiran","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sutskever, 2013] Ilya Sutskever. Training recurrent neural networks. University of Toronto Toronto, Ontario, Canada, 2013.","title":"Ilya Sutskever. Training recurrent neural networks","authors":"Sutskever","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tang and Monteleoni, 2016] Cheng Tang and Claire Mon- teleoni. convergence rate of stochastic k-means. arXiv preprint arXiv:1610.04900, 2016. [Wikipedia contributors, 2019] Wikipedia contributors.","title":"convergence rate of stochastic k-means","authors":"Cheng Tang, Claire Monteleoni","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Single-linkage clustering -Wikipedia, the free encyclo- pedia, 2019. [Online; accessed 21-January-2020].","title":"Single-linkage clustering -Wikipedia, the free encyclopedia","authors":null,"year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Zhang et al., 2017] Cheng Zhang, Hedvig Kjellstrom, and Stephan Mandt. Determinantal point processes for mini- batch diversification. arXiv preprint arXiv:1705.00607, 2017.","title":"Determinantal point processes for minibatch diversification","authors":"Cheng Zhang, Hedvig Kjellstrom, Stephan Mandt","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Zhang et al., 2019] Cheng Zhang, Cengiz Öztireli, Stephan Mandt, and Giampiero Salvi. Active mini-batch sampling using repulsive point processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5741-5748, 2019.","title":"Active mini-batch sampling using repulsive point processes","authors":"Cheng Zhang, Cengiz Öztireli, Stephan Mandt, Giampiero Salvi","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhao and Zhang, 2014] Peilin Zhao and Tong Zhang. Ac- celerating minibatch stochastic gradient descent using stratified sampling. arXiv preprint arXiv:1405.3080, 2014.","title":"Accelerating minibatch stochastic gradient descent using stratified sampling","authors":"Peilin Zhao, Tong Zhang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhao and Zhang, 2015] Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for reg- ularized loss minimization. In international conference on machine learning, pages 1-9, 2015.","title":"Stochastic optimization with importance sampling for regularized loss minimization","authors":"Peilin Zhao, Tong Zhang","year":"2015","venue":"international conference on machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}