{"type":"ScholarlyArticle","title":"On the Convergence Properties of a K-step Averaging Stochastic Gradient Descent Algorithm for Nonconvex Optimization","authors":[{"name":"Fan Zhou","slug":"fan-zhou","given":"Fan","family":"Zhou","affiliation":"Georgia Institute of Technology"},{"name":"Guojing Cong","slug":"guojing-cong","given":"Guojing","family":"Cong","affiliation":"IBM Thomas J. Watson Research Center"}],"abstract":"We adopt and analyze a synchronous K-step averaging stochastic gradient descent algorithm which we call K-AVG  for solving large scale machine learning problems. We establish the convergence results of K-AVG for nonconvex objectives. Our analysis of K-AVG applies to many existing variants of synchronous SGD.  We explain why the K-step delay is necessary and leads to better performance than traditional parallel stochastic gradient descent which is equivalent to K-AVG with $K=1$. We also show that K-AVG scales better with the number of learners than asynchronous stochastic gradient descent (ASGD). Another advantage of K-AVG over ASGD is that it allows larger stepsizes and facilitates faster convergence. On a cluster of $128$ GPUs, K-AVG is faster than ASGD implementations and achieves better accuracies and faster convergence for training with the CIFAR-10 dataset.","doi":"10.24963/ijcai.2018/447","url":"https://ijcai.org/proceedings/2018/447","pdf":"https://ijcai.org/proceedings/2018/0447.pdf","markdown":"https://ijcai.org/proceedings/2018/447.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/447","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":447,"pages":{"first":3219,"last":3227},"keywords":["Machine Learning: Learning Theory","Machine Learning: Deep Learning","Computer Vision: Statistical Methods and Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Bottou et al., 2016] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. arXiv preprint arXiv:1606.04838, 2016.","title":"Optimization methods for large-scale machine learning","authors":"Bottou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bottou, 1998] Léon Bottou. Online learning and stochas- tic approximations. On-line learning in neural networks, 17(9):142, 1998.","title":"Online learning and stochastic approximations","authors":"Léon Bottou, Bottou","year":"1998","venue":"On-line learning in neural networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2016] Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Re- visiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981, 2016.","title":"Revisiting distributed synchronous sgd","authors":"Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, Rafal Jozefowicz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung, 1954] Kai Lai Chung. On a stochastic approxima- tion method. The Annals of Mathematical Statistics, pages 463-483, 1954.","title":"On a stochastic approximation method","authors":"Kai Lai, Chung","year":"1954","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dean et al., 2012] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural in- formation processing systems, pages 1223-1231, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dekel et al., 2012] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online predic- tion using mini-batches. Journal of Machine Learning Re- search, 13(Jan):165-202, 2012.","title":"Optimal distributed online prediction using mini-batches","authors":"Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, Lin Xiao","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghadimi and Lan, 2013] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for non- convex stochastic programming. SIAM Journal on Opti- mization, 23(4):2341-2368, 2013.","title":"Stochastic first-and zeroth-order methods for nonconvex stochastic programming","authors":"Saeed Ghadimi, Guanghui Lan","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hazan and Kale, 2014] Elad Hazan and Satyen Kale. Be- yond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. The Journal of Machine Learning Research, 15(1):2489-2512, 2014.","title":"Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization","authors":"Kale ; ] Elad Hazan, Satyen Hazan, Kale","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information pro- cessing systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lian et al., 2015] Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. In Advances in Neural Informa- tion Processing Systems, pages 2737-2745, 2015.","title":"Asynchronous parallel stochastic gradient for nonconvex optimization","authors":"Lian","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2013] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.","title":"Network in network","authors":"Lin","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Loshchilov and Hutter, 2016] Ilya Loshchilov and Frank Hutter. Sgdr: stochastic gradient descent with restarts. Learning, 10:3, 2016.","title":"Ilya Loshchilov and Frank Hutter. Sgdr: stochastic gradient descent with restarts","authors":"Hutter Loshchilov","year":"2016","venue":"Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nemirovski et al., 2009] Arkadi Nemirovski, Anatoli Judit- sky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic program- ming. SIAM Journal on optimization, 19(4):1574-1609, 2009.","title":"Robust stochastic approximation approach to stochastic programming","authors":"Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, Alexander Shapiro","year":"2009","venue":"SIAM Journal on optimization","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Recht et al., 2011] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pages 693-701, 2011.","title":"Hogwild: A lock-free approach to parallelizing stochastic gradient descent","authors":"Benjamin Recht, Christopher Re, Stephen Wright, Feng Niu","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.","title":"The annals of mathematical statistics","authors":"Monro Robbins, Herbert Robbins, Sutton Monro","year":"1951","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Robbins and Siegmund, 1971] Herbert Robbins and David Siegmund. A convergence theorem for non negative al- most supermartingales and some applications. In Optimiz- ing methods in statistics, pages 233-257. Elsevier, 1971.","title":"A convergence theorem for non negative almost supermartingales and some applications","authors":"Siegmund Robbins, Herbert Robbins, David Siegmund","year":"1971","venue":"Optimizing methods in statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sacks, 1958] Jerome Sacks. Asymptotic distribution of stochastic approximation procedures. The Annals of Math- ematical Statistics, 29(2):373-405, 1958.","title":"Asymptotic distribution of stochastic approximation procedures","authors":"Jerome Sacks, Sacks","year":"1958","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shamir and Zhang, 2013] Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In International Conference on Machine Learning, pages 71- 79, 2013.","title":"Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes","authors":"Ohad Shamir, Tong Zhang","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Smith et al., 2016] Virginia Smith, Simone Forte, Chenxin Ma, Martin Takác, Michael I Jordan, and Martin Jaggi. Cocoa: A general framework for communication- efficient distributed optimization. arXiv preprint arXiv:1611.02189, 2016.","title":"Cocoa: A general framework for communicationefficient distributed optimization","authors":"Smith","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2017] Jialei Wang, Weiran Wang, and Nathan Srebro. Memory and communication efficient distributed stochastic optimization with minibatch prox. arXiv preprint arXiv:1702.06269, 2017.","title":"Memory and communication efficient distributed stochastic optimization with minibatch prox","authors":"Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2015] Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. In Advances in Neural Information Processing Systems, pages 685-693, 2015.","title":"Deep learning with elastic averaging sgd","authors":"Zhang","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2016] Jian Zhang, Christopher De Sa, Ioannis Mitliagkas, and Christopher Ré. Parallel sgd: When does averaging help? arXiv preprint arXiv:1606.07365, 2016.","title":"Parallel sgd: When does averaging help","authors":"Jian Zhang, Christopher De Sa, Ioannis Mitliagkas, Christopher Ré","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zinkevich et al., 2010] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. Parallelized stochastic gra- dient descent. In Advances in neural information process- ing systems, pages 2595-2603, 2010.","title":"Parallelized stochastic gradient descent","authors":"Martin Zinkevich, Markus Weimer, Lihong Li, Alex J Smola","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}