{"type":"ScholarlyArticle","title":"Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent","authors":[{"name":"Shuheng Shen","slug":"shuheng-shen","given":"Shuheng","family":"Shen","affiliation":"Anhui Province Key Laboratory of Big Data Analysis and Application; School of Computer Science and Technology , University of Science and Technology of China"},{"name":"Linli Xu","slug":"linli-xu","given":"Linli","family":"Xu","affiliation":"Anhui Province Key Laboratory of Big Data Analysis and Application; School of Computer Science and Technology , University of Science and Technology of China"},{"name":"Jingchang Liu","slug":"jingchang-liu","given":"Jingchang","family":"Liu","affiliation":"Anhui Province Key Laboratory of Big Data Analysis and Application; School of Computer Science and Technology , University of Science and Technology of China"},{"name":"Xianfeng Liang","slug":"xianfeng-liang","given":"Xianfeng","family":"Liang","affiliation":"Anhui Province Key Laboratory of Big Data Analysis and Application; School of Computer Science and Technology , University of Science and Technology of China"},{"name":"Yifei Cheng","slug":"yifei-cheng","given":"Yifei","family":"Cheng","affiliation":"Anhui Province Key Laboratory of Big Data Analysis and Application; School of Data Science , University of Science and Technology of China"}],"abstract":"With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are limited significantly in practice by the communication cost, making it difficult to achieve a linear time speedup. In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoD-SGD) algorithm to run computation and communication in parallel to reduce the communication cost. We prove that CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of the hardware resources. In addition, it has a lower communication complexity and better time speedup comparing with traditional distributed SGD algorithms. Experiments on deep neural network training demonstrate the significant improvements of CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs, CoCoD-SGD is up to 2-3 x faster than traditional synchronous SGD.","doi":"10.24963/ijcai.2019/637","url":"https://ijcai.org/proceedings/2019/637","pdf":"https://ijcai.org/proceedings/2019/0637.pdf","markdown":"https://ijcai.org/proceedings/2019/637.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/637","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning Applications","track":"Main track","number":637,"pages":{"first":4582,"last":4589},"keywords":["Machine Learning: Learning Theory","Machine Learning: Deep Learning","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: a system for large-scale machine learn- ing. In OSDI, volume 16, pages 265-283, 2016.","title":"Tensorflow: a system for large-scale machine learning","authors":"Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard","year":"2016","venue":"OSDI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal and Duchi, 2011] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In Ad- vances in Neural Information Processing Systems, pages 873-881, 2011.","title":"Distributed delayed stochastic optimization","authors":"Alekh Agarwal, John C Duchi","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aji and Heafield, 2017] Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing, pages 440-445, 2017.","title":"Sparse communication for distributed gradient descent","authors":"Heafield ; Aji, Fikri Alham, Kenneth Aji, Heafield","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Alistarh et al., 2017] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. In Advances in Neural Information Pro- cessing Systems, pages 1709-1720, 2017.","title":"Qsgd: Communication-efficient sgd via gradient quantization and encoding","authors":"Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, Milan Vojnovic","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Alistarh et al., 2018] Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. In Advances in Neural Information Processing Systems, pages 5977-5987, 2018.","title":"The convergence of sparsified gradient methods","authors":"Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, Cédric Renggli","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2015] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous dis- tributed systems. arXiv preprint arXiv:1512.01274, 2015.","title":"Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems","authors":"Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2016] Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Re- visiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981, 2016.","title":"Revisiting distributed synchronous sgd","authors":"Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, Rafal Jozefowicz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dekel et al., 2012] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online predic- tion using mini-batches. Journal of Machine Learning Re- search, 13(Jan):165-202, 2012.","title":"Optimal distributed online prediction using mini-batches","authors":"Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, Lin Xiao","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghadimi and Lan, 2013] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for non- convex stochastic programming. SIAM Journal on Opti- mization, 23(4):2341-2368, 2013.","title":"Stochastic first-and zeroth-order methods for nonconvex stochastic programming","authors":"Saeed Ghadimi, Guanghui Lan","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Ky- rola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Ac- curate, large minibatch sgd: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, large minibatch sgd: training imagenet in 1 hour","authors":"Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2014] Mu Li, David G Andersen, Alexander J Smola, and Kai Yu. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems, pages 19-27, 2014.","title":"Communication efficient distributed machine learning with the parameter server","authors":"Mu Li, David G Andersen, Alexander J Smola, Kai Yu","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2018] Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. Pipe-sgd: A decentralized pipelined sgd framework for distributed deep net training. In Advances in Neural In- formation Processing Systems, pages 8056-8067, 2018.","title":"Pipe-sgd: A decentralized pipelined sgd framework for distributed deep net training","authors":"Li","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lian et al., 2015] Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. In Advances in Neural Informa- tion Processing Systems, pages 2737-2745, 2015.","title":"Asynchronous parallel stochastic gradient for nonconvex optimization","authors":"Lian","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2017] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compres- sion: Reducing the communication bandwidth for dis- tributed training. arXiv preprint arXiv:1712.01887, 2017.","title":"Deep gradient compression: Reducing the communication bandwidth for distributed training","authors":"Yujun Lin, Song Han, Huizi Mao, Yu Wang, William J Dally","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. [Potluri et al., 2013]","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Sreeram Potluri, Khaled Hami- douche, Akshay Venkatesh, Devendar Bureddy, and Dhabaleswar K Panda. Efficient inter-node mpi commu- nication using gpudirect rdma for infiniband clusters with nvidia gpus. In Parallel Processing (ICPP), 2013 42nd International Conference on, pages 80-89. IEEE, 2013.","title":"Efficient inter-node mpi communication using gpudirect rdma for infiniband clusters with nvidia gpus","authors":"Sreeram Potluri, Khaled Hamidouche, Akshay Venkatesh, Devendar Bureddy, Dhabaleswar K Panda","year":"2013","venue":"42nd International Conference on","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sergeev and Balso, 2018] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799, 2018.","title":"Horovod: fast and easy distributed deep learning in TensorFlow","authors":"Alexander Sergeev, Mike Del, Balso","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Stich et al., 2018] Sebastian U Stich, Jean-Baptiste Cordon- nier, and Martin Jaggi. Sparsified sgd with memory. In Ad- vances in Neural Information Processing Systems, pages 4452-4463, 2018.","title":"Sparsified sgd with memory","authors":"Jean-Baptiste Sebastian U Stich, Martin Cordonnier, Jaggi","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Stich, 2018] Sebastian U Stich. Local sgd con- verges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018.","title":"Local sgd converges fast and communicates little","authors":"U Sebastian, Stich","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wen et al., 2017] Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in neural information process- ing systems, pages 1509-1519, 2017.","title":"Terngrad: Ternary gradients to reduce communication in distributed deep learning","authors":"Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2018] Hao Yu, Sen Yang, and Shenghuo Zhu. Par- allel restarted sgd for non-convex optimization with faster convergence and less communication. arXiv preprint arXiv:1807.06629, 2018.","title":"Parallel restarted sgd for non-convex optimization with faster convergence and less communication","authors":"Hao Yu, Sen Yang, Shenghuo Zhu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou and Cong, 2018] Fan Zhou and Guojing Cong. On the convergence properties of a k-step averaging stochas- tic gradient descent algorithm for nonconvex optimization. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 3219-3227. International Joint Conferences on Artificial Intelligence Organization, 7 2018.","title":"On the convergence properties of a k-step averaging stochastic gradient descent algorithm for nonconvex optimization","authors":"Cong Zhou, Fan Zhou, Guojing Cong","year":"2018","venue":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18","doi":null,"url":null,"internal_url":"/proceedings/2018/447"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}