{"type":"ScholarlyArticle","title":"The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication (Extended Abstract)","authors":[{"name":"Blake Woodworth","slug":"blake-woodworth","given":"Blake","family":"Woodworth","affiliation":"Toyota Technological Institute at Chicago"},{"name":"Brian Bullins","slug":"brian-bullins","given":"Brian","family":"Bullins","affiliation":"Toyota Technological Institute at Chicago"},{"name":"Ohad Shamir","slug":"ohad-shamir","given":"Ohad","family":"Shamir","affiliation":"Weizmann Institute of Science"},{"name":"Nathan Srebro","slug":"nathan-srebro","given":"Nathan","family":"Srebro","affiliation":"Toyota Technological Institute at Chicago"}],"abstract":"We resolve the min-max complexity of distributed stochastic convex optimization (up to a log factor) in the intermittent communication setting, where M machines work in parallel over the course of R rounds of communication to optimize the objective, and during each round of communication, each machine may sequentially compute K stochastic gradient estimates. We present a novel lower bound with a matching upper bound that establishes an optimal algorithm.","doi":"10.24963/ijcai.2022/751","url":"https://ijcai.org/proceedings/2022/751","pdf":"https://ijcai.org/proceedings/2022/0751.pdf","markdown":"https://ijcai.org/proceedings/2022/751.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/751","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":751,"pages":{"first":5359,"last":5363},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987264","length_sec":75,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985407","length_sec":640,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Arjevani and Shamir, 2015] Yossi Arjevani and Ohad Shamir. Communication complexity of distributed con- vex learning and optimization. In Advances in Neural Information Processing Systems, pages 1756-1764, 2015.","title":"Communication complexity of distributed convex learning and optimization","authors":"Shamir Arjevani, Yossi Arjevani, Ohad Shamir","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arjevani et al., 2019] Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Wood- worth. Lower bounds for non-convex stochastic optimiza- tion. arXiv preprint arXiv:1912.02365, 2019.","title":"Lower bounds for non-convex stochastic optimization","authors":"Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, Blake Woodworth","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Braverman et al., 2016] Mark Braverman, Ankit Garg, Tengyu Ma, Huy L Nguyen, and David P Woodruff. Com- munication lower bounds for statistical estimation prob- lems via a distributed data processing inequality. In Pro- ceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 1011-1020, 2016.","title":"Communication lower bounds for statistical estimation problems via a distributed data processing inequality","authors":"Mark Braverman, Ankit Garg, Tengyu Ma, Huy L Nguyen, David P Woodruff","year":"2016","venue":"Proceedings of the forty-eighth annual ACM symposium on Theory of Computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bullins et al., 2021] Brian Bullins, Kshitij Patel, Ohad Shamir, Nathan Srebro, and Blake E Woodworth. A stochastic newton algorithm for distributed convex opti- mization. Advances in Neural Information Processing Sys- tems, 34, 2021.","title":"A stochastic newton algorithm for distributed convex optimization","authors":"Brian Bullins, Kshitij Patel, Ohad Shamir, Nathan Srebro, Blake E Woodworth","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carmon et al., 2017] Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding sta- tionary points i. arXiv preprint arXiv:1710.11606, 2017.","title":"Lower bounds for finding stationary points i","authors":"Yair Carmon, John C Duchi, Oliver Hinder, Aaron Sidford","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chadha et al., 2021] Karan Chadha, Gary Cheng, and John C Duchi. Accelerated, optimal, and parallel: Some results on model-based stochastic optimization. arXiv preprint arXiv:2101.02696, 2021.","title":"Accelerated, optimal, and parallel: Some results on model-based stochastic optimization","authors":"Karan Chadha, Gary Cheng, John C Duchi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cotter et al., 2011] Andrew Cotter, Ohad Shamir, Nati Sre- bro, and Karthik Sridharan. Better mini-batch algorithms via accelerated gradient methods. In Advances in Neu- ral Information Processing Systems 24, pages 1647-1655, 2011.","title":"Better mini-batch algorithms via accelerated gradient methods","authors":"Andrew Cotter, Ohad Shamir, Nati Srebro, Karthik Sridharan","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dekel et al., 2012] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online predic- tion using mini-batches. Journal of Machine Learning Re- search, 13(Jan):165-202, 2012.","title":"Optimal distributed online prediction using mini-batches","authors":"Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, Lin Xiao","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hendrikx et al., 2020] Hadrien Hendrikx, Lin Xiao, Se- bastien Bubeck, Francis Bach, and Laurent Massoulie. Statistically preconditioned accelerated gradient method for distributed optimization. In International Conference on Machine Learning, pages 4203-4227. PMLR, 2020.","title":"Statistically preconditioned accelerated gradient method for distributed optimization","authors":"Hadrien Hendrikx, Lin Xiao, Sebastien Bubeck, Francis Bach, Laurent Massoulie","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lan, 2012] Guanghui Lan. An optimal method for stochas- tic composite optimization. Mathematical Programming, 133(1-2):365-397, 2012.","title":"An optimal method for stochastic composite optimization","authors":"Lan ;, Guanghui Lan","year":"2012","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nemirovskij and Yudin, 1983] Arkadij Semenovič Ne- mirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. Wiley-Interscience, 1983.","title":"Arkadij Semenovič Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization","authors":"Yudin Nemirovskij","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nesterov, 2004] Yurii Nesterov. Introductory lectures on convex optimization: A basic course. Kluwer Academic Publishers, 2004.","title":"Introductory lectures on convex optimization: A basic course","authors":"Yurii Nesterov","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nguyen et al., 2019] Lam M Nguyen, Phuong Ha Nguyen, Peter Richtárik, Katya Scheinberg, Martin Takác, and Marten van Dijk. New convergence aspects of stochas- tic gradient algorithms. Journal of Machine Learning Re- search, 20(176):1-49, 2019.","title":"New convergence aspects of stochastic gradient algorithms","authors":"Phuong Ha Lam M Nguyen, Peter Nguyen, Katya Richtárik, Martin Scheinberg, Marten Takác, Van Dijk","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shamir and Srebro, 2014] O. Shamir and N. Srebro. Dis- tributed stochastic optimization and learning. In 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 850-857, 2014.","title":"Distributed stochastic optimization and learning","authors":"O Shamir, N Srebro","year":"2014","venue":"52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Stich, 2018] Sebastian U Stich. Local sgd con- verges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018.","title":"Local sgd converges fast and communicates little","authors":"U Sebastian, Stich","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2017] Jialei Wang, Weiran Wang, and Nathan Srebro. Memory and communication efficient distributed stochastic optimization with minibatch prox. In Confer- ence on Learning Theory, pages 1882-1919. PMLR, 2017.","title":"Memory and communication efficient distributed stochastic optimization with minibatch prox","authors":"Wang","year":"2017","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Woodworth and Srebro, 2016] Blake Woodworth and Nati Srebro. Tight complexity bounds for optimizing compos- ite objectives. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural In- formation Processing Systems 29, pages 3639-3647.","title":"Tight complexity bounds for optimizing composite objectives","authors":"Srebro Woodworth, Blake Woodworth, Nati Srebro","year":"2016","venue":"Advances in Neural Information Processing Systems 29","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Woodworth et al., 2018] Blake Woodworth, Jialei Wang, Brendan McMahan, and Nathan Srebro. Graph oracle models, lower bounds, and gaps for parallel stochastic op- timization. arXiv preprint arXiv:1805.10222, 2018.","title":"Graph oracle models, lower bounds, and gaps for parallel stochastic optimization","authors":"Woodworth","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Woodworth et al., 2020] Blake Woodworth, Kumar Kshitij Patel, Sebastian U Stich, Zhen Dai, Brian Bullins, H Bren- dan McMahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? arXiv preprint arXiv:2002.07839, 2020.","title":"Is local sgd better than minibatch sgd","authors":"Blake Woodworth, Kshitij Kumar, Sebastian U Patel, Zhen Stich, Brian Dai, Brendan Bullins, Ohad Mcmahan, Nathan Shamir, Srebro","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Woodworth et al., 2021] Blake E Woodworth, Brian Bullins, Ohad Shamir, and Nathan Srebro. The min-max complexity of distributed stochastic convex optimization with intermittent communication. In Conference on Learning Theory, pages 4386-4437. PMLR, 2021.","title":"The min-max complexity of distributed stochastic convex optimization with intermittent communication","authors":"Brian Blake E Woodworth, Ohad Bullins, Nathan Shamir, Srebro","year":"2021","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yuan and Ma, 2020] Honglin Yuan and Tengyu Ma. Fed- erated accelerated stochastic gradient descent. arXiv preprint arXiv:2006.08950, 2020.","title":"Federated accelerated stochastic gradient descent","authors":"Ma ; Yuan, Honglin Yuan, Tengyu Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang and Xiao, 2015] Yuchen Zhang and Lin Xiao. Disco: Distributed optimization for self-concordant empirical loss. In International Conference on Machine Learning, pages 362-370. PMLR, 2015.","title":"Disco: Distributed optimization for self-concordant empirical loss","authors":"Yuchen Zhang, Lin Xiao","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2013a] Yuchen Zhang, John C Duchi, Michael I Jordan, and Martin J Wainwright. Information- theoretic lower bounds for distributed statistical estimation with communication constraints. In NIPS, pages 2328- 2336. Citeseer, 2013.","title":"Informationtheoretic lower bounds for distributed statistical estimation with communication constraints","authors":"Zhang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2013b] Yuchen Zhang, John C Duchi, and Martin J Wainwright. Communication-efficient algorithms for statistical optimization. The Journal of Machine Learn- ing Research, 14(1):3321-3363, 2013.","title":"Communication-efficient algorithms for statistical optimization","authors":"Zhang","year":"2013","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zinkevich et al., 2010] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. Parallelized stochastic gra- dient descent. In Advances in neural information process- ing systems, pages 2595-2603, 2010.","title":"Parallelized stochastic gradient descent","authors":"Martin Zinkevich, Markus Weimer, Lihong Li, Alex J Smola","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}