{"type":"ScholarlyArticle","title":"Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study","authors":[{"name":"Suyog Gupta","slug":"suyog-gupta","given":"Suyog","family":"Gupta","affiliation":"IBM T. J. Watson Research Center Yorktown Heights , NY 10598 , USA"},{"name":"Wei Zhang","slug":"wei-zhang","given":"Wei","family":"Zhang","affiliation":"IBM T. J. Watson Research Center Yorktown Heights , NY 10598 , USA"},{"name":"Fei Wang","slug":"fei-wang","given":"Fei","family":"Wang","affiliation":"Weill Cornell Medical College New York City , NY 10065 , USA"}],"abstract":"Deep learning with a large number of parame-ters requires distributed training, where model accuracy and runtime are two important factors to be considered. However, there has been no systematic study of the tradeoff between these two factors during the model training process. This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtime performance and model accuracy. We introduce a new learningrate modulation strategy to counter the effect of stale gradients and propose a new synchronization protocol that can effectively bound the staleness in gradients, improve runtime performance and achieve good model accuracy. Our empirical investigation reveals a principled approach for distributed training of neural networks: the mini-batch size per learner should be reduced as more learners are added to the system to preserve the model accuracy. We validate this approach using commonly-used image classification benchmarks: CIFAR10 and ImageNet.","doi":"10.24963/ijcai.2017/681","url":"https://ijcai.org/proceedings/2017/681","pdf":"https://ijcai.org/proceedings/2017/0681.pdf","markdown":"https://ijcai.org/proceedings/2017/681.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/681","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":681,"pages":{"first":4854,"last":4858},"keywords":["Artificial Intelligence: machine learning"],"videos":[],"references":[{"position":1,"raw":"Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project Adam: Building an efficient and scalable deep learning training system. OSDI'14, pages 571-582, 2014.","title":"Project Adam: Building an efficient and scalable deep learning training system","authors":"Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, Karthik Kalyanaraman","year":"2014","venue":"OSDI'14","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Adam Coates, Brody Huval, Tao Wang, David Wu, Bryan Catan- zaro, and Ng Andrew. Deep learning with cots hpc systems. In Proceedings of the 30th ICML, pages 1337-1345, 2013.","title":"Deep learning with cots hpc systems","authors":"Adam Coates, Brody Huval, Tao Wang, David Wu, Bryan Catanzaro, Ng Andrew","year":"2013","venue":"Proceedings of the 30th ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, MarcAurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In NIPS, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg S Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V Le, Mark Z Mao, Marcaurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Andrew Y Ng","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Olga Russakovsky et al. ImageNet Large Scale Visual Recognition Challenge. IJCV, pages 1-42, 2015.","title":"ImageNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature em- bedding. arXiv preprint arXiv:1408.5093, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, Trevor Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Computer Science Department, Uni- versity of Toronto, Tech. Rep, 1(4):7, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":"Tech. Rep","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Ad- vances in neural information processing systems, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Wei Zhang, Suyog Gupta, Xiangru Lin, Ji Liu, and Fei Wang. Model accuracy and runtime tradeoff in distributed deep learning: A sys- tematic study and its theoretical underpinning. Knowledge and Information Systems, (In submission), 2017.","title":"Model accuracy and runtime tradeoff in distributed deep learning: A systematic study and its theoretical underpinning","authors":"Wei Zhang, Suyog Gupta, Xiangru Lin, Ji Liu, Fei Wang","year":"2017","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}