{"type":"ScholarlyArticle","title":"Differential Equations for Modeling Asynchronous Algorithms","authors":[{"name":"Li He","slug":"li-he","given":"Li","family":"He","affiliation":"University of Chinese Academy of Sciences; Academy of Mathematics and Systems Science , Chinese Academy of Sciences"},{"name":"Qi Meng","slug":"qi-meng","given":"Qi","family":"Meng","affiliation":"Peking University"},{"name":"Wei Chen","slug":"wei-chen","given":"Wei","family":"Chen","affiliation":"Microsoft Research"},{"name":"Zhi-Ming Ma","slug":"zhi-ming-ma","given":"Zhi-Ming","family":"Ma","affiliation":"University of Chinese Academy of Sciences; Academy of Mathematics and Systems Science , Chinese Academy of Sciences"},{"name":"Tie-Yan Liu","slug":"tie-yan-liu","given":"Tie-Yan","family":"Liu","affiliation":"Microsoft Research"}],"abstract":"Asynchronous stochastic gradient descent (ASGD) is a popular parallel optimization algorithm in machine learning. Most theoretical analysis on ASGD take a discrete view and prove upper bounds for their convergence rates. However, the discrete view has its intrinsic limitations: there is no characterizationof the optimization path and the proof techniques are induction-based and thus usually complicated. Inspired by the recent successful adoptions of stochastic differential equations (SDE) to the theoretical analysis of SGD, in this paper, we study the continuous approximation of ASGD by using stochastic differential delay equations (SDDE). We introduce the approximation method and study the approximation error. Then we conduct theoretical analysis on the convergence rate of ASGD algorithm based on the continuous approximation.There are two methods: moment estimation and energy function minimization can be used to analyzethe convergence rates. Moment estimation depends on the specific form of the loss function, while energy function minimization only leverages the convex property of the loss function, and does not depend on its specific form. In addition to the convergence analysis, the continuous view also helps us derive better convergence rates. All of this clearly shows the advantage of taking the continuous view in gradient descent algorithms.","doi":"10.24963/ijcai.2018/307","url":"https://ijcai.org/proceedings/2018/307","pdf":"https://ijcai.org/proceedings/2018/0307.pdf","markdown":"https://ijcai.org/proceedings/2018/307.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/307","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":307,"pages":{"first":2220,"last":2226},"keywords":["Machine Learning: Learning Theory","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Agarwal and Duchi, 2011] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In Ad- vances in Neural Information Processing Systems, pages 873-881, 2011.","title":"Distributed delayed stochastic optimization","authors":"Alekh Agarwal, John C Duchi","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balles et al., 2017] Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learn- ing rates. Conference on Uncertainty in Artificial Intelli- gence, 2017.","title":"Coupling adaptive batch sizes with learning rates","authors":"Lukas Balles, Javier Romero, Philipp Hennig","year":"2017","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bao et al., 2014] Jianhai Bao, George Yin, and Chenggui Yuan. Ergodicity for functional stochastic differential e- quations and applications. Nonlinear Analysis: Theory, Methods & Applications, 98:66-82, 2014.","title":"Ergodicity for functional stochastic differential equations and applications","authors":"Bao","year":"2014","venue":"Nonlinear Analysis: Theory, Methods & Applications","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bao et al., 2016] Jianhai Bao, George Yin, and Chenggui Yuan. Asymptotic analysis for functional stochastic dif- ferential equations, 2016.","title":"Asymptotic analysis for functional stochastic differential equations","authors":"Bao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dean et al., 2012] Jeffrey Dean, Greg Corrado, Rajat Mon- ga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale dis- tributed deep networks. In Advances in neural information processing systems, pages 1223-1231, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Durrett, 2010] Rick Durrett. Probability: theory and exam- ples. Cambridge university press, 2010.","title":"Rick Durrett. Probability: theory and examples","authors":"Durrett","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krichene et al., 2015] W Krichene, Am Bayen, and Pl Bartlett. Accelerated mirror descent in continuous and discrete time. pages 2845-2853, 2015.","title":"Accelerated mirror descent in continuous and discrete time","authors":"Am Krichene, Pl Bayen, Bartlett","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Langford et al., 2009] John Langford, Alex J Smola, and Martin Zinkevich. Slow learners are fast. Advances in Neural Information Processing Systems, 22:2331-2339, 2009.","title":"Slow learners are fast","authors":"John Langford, Alex J Smola, Martin Zinkevich","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2016] Chris Junchi Li, Zhaoran Wang, and Han Li- u. Online ica: Understanding global dynamics of non- convex optimization via diffusion processes. In Advances in Neural Information Processing Systems, pages 4961- 4969, 2016.","title":"Online ica: Understanding global dynamics of nonconvex optimization via diffusion processes","authors":"Chris Junchi Li, Zhaoran Wang, Han Liu","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017] Qianxiao Li, Cheng Tai, and Weinan E. S- tochastic modified equations and adaptive stochastic gra- dient algorithms. In Proceedings of the 34th Internation- al Conference on Machine Learning, pages 2101-2110, 2017.","title":"Stochastic modified equations and adaptive stochastic gradient algorithms","authors":"Li","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lian et al., 2015] Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradien- t for nonconvex optimization. In International Conference on Neural Information Processing Systems, pages 2737- 2745, 2015.","title":"Asynchronous parallel stochastic gradient for nonconvex optimization","authors":"Lian","year":"2015","venue":"International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mandt et al., 2016] Stephan Mandt, Matthew D Hoffman, and David M Blei. A variational analysis of stochastic gradient algorithms. In Proceedings of The 33rd Interna- tional Conference on Machine Learning, pages 354-363, 2016.","title":"A variational analysis of stochastic gradient algorithms","authors":"Stephan Mandt, Matthew D Hoffman, David M Blei","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mao, 2007] Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.","title":"Xuerong Mao. Stochastic differential equations and applications","authors":"Mao","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Raginsky and Bouvrie, 2012] Maxim Raginsky and J Bou- vrie. Continuous-time stochastic mirror descent on a network: Variance reduction, consensus, convergence. 23(1):6793-6800, 2012.","title":"Maxim Raginsky and J Bouvrie. Continuous-time stochastic mirror descent on a network: Variance reduction, consensus","authors":"Bouvrie Raginsky","year":"2012","venue":"convergence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rakhlin et al., 2011] Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.5647, 2011.","title":"Making gradient descent optimal for strongly convex stochastic optimization","authors":"Alexander Rakhlin, Ohad Shamir, Karthik Sridharan","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Recht et al., 2011] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, pages 693-701, 2011.","title":"Hogwild: A lock-free approach to parallelizing stochastic gradient descent","authors":"Benjamin Recht, Christopher Re, Stephen Wright, Feng Niu","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Su et al., 2014] Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nesterov's accelerated gradient method: theory and insights. In Ad- vances in Neural Information Processing Systems, pages 2510-2518, 2014.","title":"A differential equation for modeling nesterov's accelerated gradient method: theory and insights","authors":"Weijie Su, Stephen Boyd, Emmanuel Candes","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Uhlenbeck and Ornstein, 1930] George E Uhlenbeck and Leonard S Ornstein. On the theory of the brownian mo- tion. Physical review, 36(5):823, 1930.","title":"On the theory of the brownian motion","authors":"Ornstein ; Uhlenbeck, E George, Leonard S Uhlenbeck, Ornstein","year":"1930","venue":"Physical review","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wibisono et al., 2016] Andre Wibisono, Ashia C Wilson, and Michael I Jordan. A variational perspective on acceler- ated methods in optimization. Proceedings of the National Academy of Sciences, page 201614734, 2016.","title":"A variational perspective on accelerated methods in optimization","authors":"Andre Wibisono, Michael I Ashia C Wilson, Jordan","year":"2016","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2017] Lin F. Yang, R. Arora, V. Braverman, and Tuo Zhao. The physical systems behind optimization al- gorithms. 2017.","title":"The physical systems behind optimization algorithms","authors":"Lin F Yang, R Arora, V Braverman, Tuo Zhao","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2015] Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. In Advances in Neural Information Processing Systems, pages 685-693, 2015.","title":"Deep learning with elastic averaging sgd","authors":"Zhang","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zheng et al., 2017] Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Li- u. Asynchronous stochastic gradient descent with delay compensation. In International Conference on Machine Learning, pages 4120-4129, 2017.","title":"Asynchronous stochastic gradient descent with delay compensation","authors":"Zheng","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}