{"type":"ScholarlyArticle","title":"A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification","authors":[{"name":"Shaohuai Shi","slug":"shaohuai-shi","given":"Shaohuai","family":"Shi","affiliation":"Department of Computer Science , Hong Kong Baptist University"},{"name":"Kaiyong Zhao","slug":"kaiyong-zhao","given":"Kaiyong","family":"Zhao","affiliation":"Department of Computer Science , Hong Kong Baptist University"},{"name":"Qiang Wang","slug":"qiang-wang","given":"Qiang","family":"Wang","affiliation":"Department of Computer Science , Hong Kong Baptist University"},{"name":"Zhenheng Tang","slug":"zhenheng-tang","given":"Zhenheng","family":"Tang","affiliation":"Department of Computer Science , Hong Kong Baptist University"},{"name":"Xiaowen Chu","slug":"xiaowen-chu","given":"Xiaowen","family":"Chu","affiliation":"Department of Computer Science , Hong Kong Baptist University"}],"abstract":"Gradient sparsification is a promising technique to significantly reduce the communication overhead in decentralized synchronous stochastic gradient descent (S-SGD) algorithms. Yet, many existing gradient sparsification schemes (e.g., Top-k sparsification) have a communication complexity of O(kP), where k is the number of selected gradients by each worker and P is the number of workers. Recently, the gTop-k sparsification scheme has been proposed to reduce the communication complexity from O(kP) to O(k logP), which significantly boosts the system scalability. However, it remains unclear whether the gTop-k sparsification scheme can converge in theory. In this paper, we first provide theoretical proofs on the convergence of the gTop-k scheme for non-convex objective functions under certain analytic assumptions. We then derive the convergence rate of gTop-k S-SGD, which is at the same order as the vanilla mini-batch SGD. Finally, we conduct extensive experiments on different machine learning models and data sets to verify the soundness of the assumptions and theoretical results, and discuss the impact of the compression ratio on the convergence performance.","doi":"10.24963/ijcai.2019/473","url":"https://ijcai.org/proceedings/2019/473","pdf":"https://ijcai.org/proceedings/2019/0473.pdf","markdown":"https://ijcai.org/proceedings/2019/473.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/473","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":473,"pages":{"first":3411,"last":3417},"keywords":["Machine Learning Applications: Big data ; Scalability","Computer Vision: Big Data and Large Scale Methods","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Acero, 1990] Alejandro Acero. Acoustical and environ- mental robustness in automatic speech recognition. In ICASSP, 1990.","title":"Acoustical and environmental robustness in automatic speech recognition","authors":"Alejandro Acero","year":"1990","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aji and Heafield, 2017] Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. In EMNLP, pages 440-445, 2017.","title":"Sparse communication for distributed gradient descent","authors":"Heafield ; Aji, Fikri Alham, Kenneth Aji, Heafield","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alistarh et al., 2017] Dan Alistarh, Demjan Grubic, Jerry Li, et al. QSGD: Communication-efficient SGD via gradi- ent quantization and encoding. In NIPS, pages 1709-1720, 2017.","title":"QSGD: Communication-efficient SGD via gradient quantization and encoding","authors":"Dan Alistarh, Demjan Grubic, Jerry Li","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Alistarh et al., 2018] Dan Alistarh, Torsten Hoefler, Mikael Johansson, et al. The convergence of sparsified gradient methods. In NeurIPS, pages 5977-5987, 2018.","title":"The convergence of sparsified gradient methods","authors":"Dan Alistarh, Torsten Hoefler, Mikael Johansson","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Awan et al., 2017] Ammar Ahmad Awan, Khaled Hami- douche, Jahanzeb Maqbool Hashmi, et al. S-Caffe: Co- designing MPI runtimes and Caffe for scalable deep learn- ing on modern GPU clusters. In PPoPP, pages 193-205. ACM, 2017.","title":"Codesigning MPI runtimes and Caffe for scalable deep learning on modern GPU clusters","authors":"Ammar Ahmad Awan, Khaled Hamidouche, Jahanzeb Maqbool Hashmi","year":"2017","venue":"PPoPP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bernstein et al., 2018] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, et al. signSGD: compressed op- timisation for non-convex problems. In ICML, pages 559- 568, 2018.","title":"signSGD: compressed optimisation for non-convex problems","authors":"Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bottou et al., 2018] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.","title":"Optimization methods for large-scale machine learning","authors":"Bottou","year":"2018","venue":"Siam Review","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2018] Chia-Yu Chen, Jungwook Choi, Daniel Brand, et al. Adacomp: Adaptive residual gradient com- pression for data-parallel distributed training. In AAAI, pages 2827-2835, 2018.","title":"Adacomp: Adaptive residual gradient compression for data-parallel distributed training","authors":"Chia-Yu Chen, Jungwook Choi, Daniel Brand","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2019] Chen Chen, Wei Wang, and Bo Li. Round-robin synchronization: Mitigating communication bottlenecks in parameter servers. In IEEE INFOCOM, 2019.","title":"Round-robin synchronization: Mitigating communication bottlenecks in parameter servers","authors":"Chen Chen, Wei Wang, Bo Li","year":"2019","venue":"IEEE INFOCOM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dean et al., 2012] Jeffrey Dean, Greg Corrado, Rajat Monga, et al. Large scale distributed deep networks. In NIPS, pages 1223-1231, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg Corrado, Rajat Monga","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dekel et al., 2012] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, et al. Optimal distributed online prediction us- ing mini-batches. Journal of Machine Learning Research, 13(Jan):165-202, 2012.","title":"Optimal distributed online prediction using mini-batches","authors":"Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, et al. Accurate, large minibatch SGD: training Ima- geNet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, large minibatch SGD: training Ima-geNet in 1 hour","authors":"Priya Goyal, Piotr Dollár, Ross Girshick","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Haddadpour et al., 2019] Farzin Haddadpour, Moham- mad Mahdi Kamani, Mehrdad Mahdavi, et al. Trading redundancy for communication: Speeding up distributed SGD for non-convex optimization. In ICML, pages 2545-2554, 2019.","title":"Trading redundancy for communication: Speeding up distributed SGD for non-convex optimization","authors":"Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jia et al., 2018] Xianyan Jia, Shutao Song, Shaohuai Shi, et al. Highly scalable deep learning training system with mixed-precision: Training ImageNet in four minutes. In NeurIPS Workshop MLSys, 2018.","title":"Highly scalable deep learning training system with mixed-precision: Training ImageNet in four minutes","authors":"Jia","year":"2018","venue":"NeurIPS Workshop MLSys","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jiang and Agrawal, 2018] Peng Jiang and Gagan Agrawal. A linear speedup analysis of distributed deep learning with sparse and quantized communication. In NeurIPS, pages 2530-2541, 2018.","title":"A linear speedup analysis of distributed deep learning with sparse and quantized communication","authors":"Peng Jiang, Gagan Agrawal","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2018] Yujun Lin, Song Han, Huizi Mao, et al. Deep gradient compression: Reducing the communication bandwidth for distributed training. In ICLR, 2018.","title":"Deep gradient compression: Reducing the communication bandwidth for distributed training","authors":"Yujun Lin, Song Han, Huizi Mao","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Marcus et al., 1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2):313-330, 1993.","title":"Building a large annotated corpus of English: The Penn Treebank","authors":"Mary Mitchell P Marcus, Ann Marcinkiewicz, Beatrice Santorini","year":"1993","venue":"Computational linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Renggli et al., 2018] Cèdric Renggli, Dan Alistarh, and Torsten Hoefler. SparCML: High-performance sparse communication for machine learning. arXiv preprint arXiv:1802.08021, 2018.","title":"SparCML: High-performance sparse communication for machine learning","authors":"Renggli","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shi et al., 2018] Shaohuai Shi, Wang Qiang, and Xiaowen Chu. Performance modeling and evaluation of distributed deep learning frameworks on GPUs. In IEEE DataCom, pages 949-957, 2018.","title":"Performance modeling and evaluation of distributed deep learning frameworks on GPUs","authors":"Shi","year":"2018","venue":"IEEE DataCom","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2019a] Shaohuai Shi, Xiaowen Chu, and Bo Li. MG-WFBP: Efficient data communication for distributed synchronous SGD algorithms. In IEEE INFOCOM, 2019.","title":"MG-WFBP: Efficient data communication for distributed synchronous SGD algorithms","authors":"Shi","year":"2019","venue":"IEEE INFOCOM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shi et al., 2019b] Shaohuai Shi, Qiang Wang, Kaiyong Zhao, et al. A distributed synchronous SGD algorithm with global Top-k sparsification for low bandwidth net- works. In IEEE ICDCS, 2019.","title":"A distributed synchronous SGD algorithm with global Top-k sparsification for low bandwidth networks","authors":"Shi","year":"2019","venue":"IEEE ICDCS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Stich et al., 2018] Sebastian U Stich, Jean-Baptiste Cordon- nier, and Martin Jaggi. Sparsified SGD with memory. In NeurIPS, pages 4452-4463, 2018.","title":"Sparsified SGD with memory","authors":"Jean-Baptiste Sebastian U Stich, Martin Cordonnier, Jaggi","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Hongyi Wang, Scott Sievert, Shengchao Liu, et al. Atomo: Communication-efficient learning via atomic sparsification. In NeurIPS, pages 9850-9861, 2018.","title":"Atomo: Communication-efficient learning via atomic sparsification","authors":"Hongyi Wang, Scott Sievert, Shengchao Liu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wangni et al., 2018] Jianqiao Wangni, Jialei Wang, Ji Liu, et al. Gradient sparsification for communication-efficient distributed optimization. In NeurIPS, pages 1306-1316, 2018.","title":"Gradient sparsification for communication-efficient distributed optimization","authors":"Wangni","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wen et al., 2017] Wei Wen, Cong Xu, Feng Yan, et al. Tern- grad: Ternary gradients to reduce communication in dis- tributed deep learning. In NIPS, pages 1509-1519, 2017.","title":"Terngrad: Ternary gradients to reduce communication in distributed deep learning","authors":"Wei Wen, Cong Xu, Feng Yan","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2018] Jiaxiang Wu, Weidong Huang, Junzhou Huang, et al. Error compensated quantized SGD and its applications to large-scale distributed optimization. In ICML, pages 5321-5329, 2018.","title":"Error compensated quantized SGD and its applications to large-scale distributed optimization","authors":"Jiaxiang Wu, Weidong Huang, Junzhou Huang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017] Hao Zhang, Zeyu Zheng, Shizhen Xu, et al. Poseidon: an efficient communication architecture for distributed deep learning on GPU clusters. In USENIX ATC, pages 181-193, 2017.","title":"Poseidon: an efficient communication architecture for distributed deep learning on GPU clusters","authors":"Hao Zhang, Zeyu Zheng, Shizhen Xu","year":"2017","venue":"USENIX ATC","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}