--- title: "Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study" authors: - "Suyog Gupta" - "Wei Zhang" - "Fei Wang" doi: "10.24963/ijcai.2017/681" url: "https://ijcai.org/proceedings/2017/681" pdf: "https://ijcai.org/proceedings/2017/0681.pdf" venue: "Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence" year: 2017 pages: "4854-4858" track: "Best Sister Conferences" keywords: - "Artificial Intelligence: machine learning" --- # Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study Suyog Gupta, Wei Zhang, Fei Wang *Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence*, pages 4854-4858. DOI: [10.24963/ijcai.2017/681](https://doi.org/10.24963/ijcai.2017/681) ## Abstract Deep learning with a large number of parame-ters requires distributed training, where model accuracy and runtime are two important factors to be considered. However, there has been no systematic study of the tradeoff between these two factors during the model training process. This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtime performance and model accuracy. We introduce a new learningrate modulation strategy to counter the effect of stale gradients and propose a new synchronization protocol that can effectively bound the staleness in gradients, improve runtime performance and achieve good model accuracy. Our empirical investigation reveals a principled approach for distributed training of neural networks: the mini-batch size per learner should be reduced as more learners are added to the system to preserve the model accuracy. We validate this approach using commonly-used image classification benchmarks: CIFAR10 and ImageNet. ## Full text ## 1 Introduction Deep neural network based models have achieved unparalleled accuracy in cognitive tasks such as speech recognition, object detection, and natural language processing LeCun et al. [2015]. Recent years have seen a resurgence of interest in deploying large-scale computing infrastructure designed specifically for training deep neural networks. Some notable efforts in this direction include distributed computing infrastructure using thousands of CPU cores Chilimbi et al. [2014]; Dean et al. [2012], high-end graphics processors (GPUs) Krizhevsky et al. [2012], or a combination of CPUs and GPUs Coates et al. [2013]. Contrary to the popular belief, among the system researchers, that asynchrony necessarily improves model accuracy, we find that adopting the approach of scale-out deep ## Fei Wang Weill Cornell Medical College New York City, NY 10065, USA feiwang03@gmail.com learning using asynchronous-SGD, gives rise to complex interdependencies between the training algorithm's hyperparameters and the distributed implementation's design choices (synchronization protocol, number of learners), ultimately impacting the neural network's accuracy and the overall system's runtime performance. In this paper we present Rudra, a parameter server based deep learning framework to study these interdependencies and undertake an empirical evaluation with public image classification benchmarks. Our key contributions are: 1. A systematic technique (vector clock) for quantifying the staleness of gradient descent parameter updates. 2. An investigation of the impact of the interdependence of training algorithm's hyperparameters (mini-batch size, learning rate (gradient descent step size)) and distributed implementation's parameters (gradient staleness, number of learners) on the neural network's classification accuracy and training time. 3. A new learning rate tuning strategy that reduces the effect of stale parameter updates. 4. A new synchronization protocol to reduce network bandwidth overheads while achieving good classification accuracy and runtime performance. 5. An observation that to maintain a given level of model accuracy, it is necessary to reduce the mini-batch size as the number of learners is increased. This suggests a hard limit on the amount of parallelism that can be exploited in training a given model. ## 2 Design and Implementation ## 2.1 Terminology - Parameter Server: a server that holds the model weights. The parameter server collects gradients from learners and updates the weights accordingly. - Learner: A computing process that can calculate weight updates (gradients). - µ: mini-batch size. - λ: number of learners. - α: learning rate. - Epoch: a pass through the entire training dataset. - Timestamp: each weight update increments the timestamp by 1. The timestamp of a gradient is the same as the timestamp of the weight used to compute the gradient. - σ: staleness of the gradient. A gradient with timestamp tsi is pushed to the parameter server with current weight timestamp tsj , where tsj ≥ tsi. We define the staleness of this gradient σ as j - i. - 〈σ〉, average staleness of gradients. The timestamps of the set of n gradients that triggers the advancement of weights timestamp from tsi-1 to tsi form a vector clock〈tsi1 , tsi2 , ..., tsin 〉, where max{i1, i2, ..., in} < i. The average staleness of gradients 〈σ〉 is defined as: - Hardsync protocol: To advance weights timestamp from tsi to tsi+1, each learner calculates exactly one minibatch and sends its gradient ∇θl to the parameter server. The parameter server averages the gradients and updates the weights according to Equation (2), then broadcasts the newweights to all learners. Staleness in the hardsync protocol is always zero. - Async protocol: Each learner calculates the gradients and asynchronously pushes/pulls the gradients/weights to/from parameter server. The Async weight update rule is given by: Gradient staleness may be hard to control due to the asynchrony in the system. Dean et al. [2012] describe Downpour SGD, an implementation of the Async protocol for a commodity scale-out system in which the staleness can be as large as hundreds. - n-softsync protocol: Each learner pulls the weights from the parameter server, calculates the gradients and pushes the gradients to the parameter server. The parameter server updates the weights after collecting at least c = ⌊(λ/n)⌋ gradients. The splitting parameter n can vary from 1 to λ. The n-softsync weight update rule is given by: In a homogeneous cluster where each learner proceeds at roughly the same speed, the staleness of the model is expected to be n. Note that when n is equal to λ, the weight update rule at the parameter server is exactly the same as in Async protocol. Figure 1: Rudra-base architecture ![Image](https://ijcai.org/proceedings/2017/fulltext/681/image_000000_b00db4cf976fcfac3945df50e73ea4bfe9fe701e8399abec49cbefab501687ed.png) Figure 2: Rudra-adv and Rudra-adv∗ architecture ![Image](https://ijcai.org/proceedings/2017/fulltext/681/image_000001_dac7553de380927413ed67f724b5f1700943d2fb25566220bfd9895bb9e379f1.png) ## 2.2 Rudra SystemArchitecture Rudra-base Figure 1 illustrates the base-line parameter server design that we use to study the interplay of hyperparameter tuning and system scale-out factor. Parameter Server is a multithreaded process, that accumulates gradients from each learner and applies update rules according to Equations (2-4) (i.e., hardsync and n-softsync protocols). Learner is a singleprocess multithreaded SGD solver, it calculates the gradients and sends the gradients along with their timestamp to the parameter server. Learner retrieves weights from the Parameter Server when newer weights are generated. Data Server hosts training and testing data. Statistics Server records training statistics (e.g., training error). Rudra-adv To alleviate the network traffic to parameter server, we build a parameter server group that forms a tree. We co-locate each tree leaf node on the same node as the learners for which it is responsible. Each node in the parameter server group is responsible for averaging the gradients sent from its learners and relaying the averaged gradient to its parent. The root node in the parameter server group is responsible for applying weight update and broadcasting the updated weights. Each non-leaf node pulls the weights from its parent and responds to its children's weight pulling requests. Figure 2(b) illustrates the system architecture for Rudra-adv. Rudra-adv∗ . We built Rudra-adv∗ to further improve the runtime performance of Rudra-adv in two ways: (1) Broadcast weights within learners to utilize network links between learners. (2) Asynchronous pushGradient and pullWeights. Figure 2(b) illustrates the system architecture for Rudra-adv∗ . Different from Rudra-adv, each learner continuously receives weights from the weights downcast tree, which consists of the top level parameter server node and all the learners. ## 3 Methodology ## 3.1 Hardware and Software Environment We deploy the Rudra distributed deep learning framework on a P775 supercomputer. Each node of this system contains four eight-core 3.84GHz POWER7 processors, one optical connect controller chip and 128GB of memory. A single node has a theoretical floating point peak performance of 982Gflop/s, memory bandwidth of 512GB/s and bidirectional interconnect bandwidth of 192GB/s.The cluster operating system is Red Hat Enterprise Linux 6.4. To compile and run Rudra we used the IBM xlC compiler version 12.1 with the -O3 -q64 -qsmp options, ESSL for BLAS subroutines, and IBMMPI (IBM Parallel Operating Environment 1.2). ## 3.2 Benchmark Datasets and Neural Network Architectures To evaluate Rudra's scale-out performance we employ two commonly used image classification benchmark datasets: CIFAR10 Krizhevsky and Hinton [2009] and ImageNet et al [2015]. The neural network architecture that we used for CIFAR10 closely mimics the CIFAR10 model (cifar10 full.prototxt) available as a part of the open-source Caffe deep learning package Jia et al. [2014]. The neural network is trained using momentum-accelerated mini-batch SGD with a batch size of 128 and momentum set to 0.9. The base learning rate α0 is set to 0.001 are reduced by a factor of 10 after the 120 th and 130 th epoch. For ImageNet, we use the neural network architecture introduced in Krizhevsky et al. [2012] consisting of 5 convolutional layers and 3 fullyconnected layers. The initial learning rate α0 is set equal to 0.01 and reduced by a factor of 10 after the 15 th and 25 th epoch. ## 4 Evaluation In this section we present results of evaluation of our scaleout deep learning training implementation. For an initial design space exploration, we use the CIFAR10 dataset and Rudra-base system architecture (Section 4.1 and Section 4.2). Subsequently we extend our findings to the ImageNet dataset using the Rudra-adv and Rudra-adv∗ system architectures (Section 4.3). ## 4.1 Staleness-Aware Learning Rate Tuning In our experiments with the n-softsync protocol we found it beneficial, and at times necessary, to modulate the learning rate α to take into account the staleness of the gradients. For the n-softsync protocol, we set the learning rate as: where α0 is the learning rate used for the baseline (control) experiment: µ = 128, λ = 1. Figure 3 shows a set of representative results illustrating the benefits of adopting this learning rate modulation strategy. We show the evolution of the test error on the CIFAR10 dataset as a function of the training epoch for two different configurations of the n-softsync protocol (n = 4, n = 30) and set the number of learners, λ = 30. In both these configurations, setting the learning rate in accordance with equation (5) results in lower test error as compared with the cases where the learning rate is set to α0. Surprisingly, the configuration 30-softsync, λ = 30, α = α0 fails to converge and shows a constant high error rate of 90%. Reducing the learning rate by a factor 〈σ〉 = n = 30 makes the model with much lower test error. Figure 3: Effect of learning rate modulation strategy: Dividing the learning rate by the average staleness aids in better convergence and achieves lower test error when using the n-softsync protocol. Number of learners, λ = 30; mini-batch size µ = 128. ![Image](https://ijcai.org/proceedings/2017/fulltext/681/image_000002_e477617d43717d4c40992326a7f40ccdd89c399a8e07b8adcec9901ad9890bef.png) ## 4.2 µλ = constant It is trivial to conclude that training with different configurations, in which µλ = constant, yields similar (if not the same) model accuracy for the hardsync protocol. Interestingly, we find that even for the n-softsync protocol, configurations that maintain µλ = constant achieve comparable test errors. In Table 2 we report the test error at the end of 140 epochs for configurations with µλ = constant. For instance, Table 2 shows that when µλ ≈ 128, the test error stays ∼18-19%. Table 2 also shows that the test error increases monotonically with the µλ product. These results reveal the scalability limits under the constraints of preserving the model accuracy. These results help define a principled approach for distributed training of neural networks: the minibatch size per learner should be reduced as more learners are added to the system in way that keeps µλ product constant. In addition, the learning rate should be modulated to account for stale gradients. From machine learning perspective, this points to an interesting research direction on designing optimization algorithm and learning strategies that perform well with large mini-batch sizes. ## 4.3 Results on ImageNet Benchmark The computational cost of training ImageNet prohibits an exhaustive state space exploration of the interplay between model accuracy and system runtime performance. The baseline configuration (µ = 256, λ = 1) takes 54 hours/epoch. Guided by the results of section 4.2, we first consider a configuration with µ = 16, λ = 18 and employ the Rudra-base architecture with hardsync protocol (base-hardsync). This configuration performs training at the speed of ∼330 minutes/epoch and achieves a top-5 error of 20.85%, matching the accuracy of the baseline configuration (µ = 256, λ = 1). Table 1: Results on ImageNet benchmark: Validation error at the end of 30 epochs and training time per epoch for different configurations. | Configuration | Architecture | µ | λ | Synchronization protocol | Validation error(top-1) | Validation error (top-5) | Training time (minutes/epoch) | |-----------------|----------------|-----|-----|----------------------------|---------------------------|----------------------------|---------------------------------| | base-hardsync | Rudra-base | 16 | 18 | Hardsync ) | 44.35% | 20.85% | 330 | | base-softsync | Rudra-base | 16 | 18 | 1-softsync % | 45.63% | 22.08% | 270 | | adv-softsync | Rudra-adv | 4 | 54 | 1-softsync ( | 46.09% | 22.44% | 212 | | adv∗ -softsync | Rudra-adv∗ | 4 | 54 | 1-softsync ) | 46.53% | 23.38% | 125 | Table 2: Results on CIFAR10 benchmark: Test error at the end of 140 epochs and training time for (σ, µ, λ) configurations with µλ = constant. | σ | µ | λ | Test error | Training time(s) | |--------------|-----|-----|----------------------|--------------------| | 1 30 | 4 4 | 30 | 18.09% 18.41% 18.92% | 1573 2073 | | | | 30 | | | | µλ ≈ 128 18 | 8 | 18 | | 2488 | | 10 | 16 | 10 | 18.79% | 3396 | | 4 | 32 | 4 | 18.82% | 7776 | | 2 | 64 | 2 | 17.96% | 13449 | | 1 | 8 | 30 | 20.04% | 1478 | | 30 | 8 | 30 | 19.65% | 1509 | | 18 | 16 | 18 | 20.33% | 2938 | | µλ ≈ 256 10 | 32 | 10 | 20.82% | 3518 | | 4 | 64 | 4 | 20.70% | 6631 | | 2 | 128 | 2 | 19.52% | 11797 | | 1 | 128 | 2 | 19.59% | 11924 | | 1 | 16 | 30 | 23.25% | 1469 | | 30 | 16 | 30 | 22.14% | 1502 | | µλ ≈ 512 18 | 32 | 18 | 23.63% | 2255 | | 10 | 64 | 10 | 24.08% | 2683 | | 4 | 128 | 4 | 23.01% | 7089 | | 1 | 32 | 30 | 27.16% | 1299 | | 30 | 32 | 30 | 27.27% | 1420 | | µλ ≈ 1024 18 | 64 | 18 | 28.31% | 1713 | | 1 | 128 | 10 | 29.83% | 2551 | | 10 | 128 | 10 | 29.90% | 2626 | Training using the 1-softsync protocol with mini-batch size of µ = 16 and 18 learners takes ∼270 minutes/epoch, reaching a top-1 (top-5) accuracy of 45.63% (22.08%) by the end of 30 epochs (base-softsync). λ = 54 learners, each processing a mini-batch size µ = 4, train at ∼212 minutes/epoch when using Rudra-adv architecture and 1-softsync protocol (adv-softsync). As in the case of Rudra-base, the average staleness in the gradients is close to 1 and this configuration also achieves a top-1(top-5) error of 46.09% (22.44%). The Rudra-adv∗ improves the runtime performance to ∼125 minutes/epoch, and its top-1 validation error is (46.53%). Table 1 summarizes the results obtained for the 4 configurations discussed above. Figure 4 compares the evolution of the top-1 validation error during training for the 4 different configuration summarized in Table 1. The training speed follows the order adv∗ -softsync > adv-softsync > base-softsync > base-hardsync. As a result, adv∗ -softsync is the first configuration to hit the 48% validation error mark. Configurations other than base-hardsync show marginally higher validation error compared with the baseline. ## 5 Conclusion In this paper, we empirically studied the interplay of hyperparameter tuning and scale-out in three protocols for communicating model weights in asynchronous stochastic gradient descent. We divide the learning rate by the average stale- b a s e - h a r d s y n ![Image](https://ijcai.org/proceedings/2017/fulltext/681/image_000003_1811fc0f6451a3c4dfb411bc58106006a7a0d186d96e0e4f4bce5eca45f3ee7a.png) T r a i n i n g t i m e ( h o u r s Figure 4: Results on ImageNet benchmark: Error on the validation set as a function of training time for the configurations listed in Table 1 ness of gradients, resulting in faster convergence and lower test error. Our experiments show that the 1-softsync protocol (in which the parameter server accumulates λ gradients before updating the weights) minimizes gradient staleness and achieves the lowest runtime for a given test error. We found that to maintain a model accuracy, it is necessary to reduce the mini-batch size as the number of learners is increased. This suggests an upper limit on the level of parallelism that can be exploited for a given model, and consequently a need for algorithms that permit training over larger batch sizes. We provide theoretical justification of these empirical results in a separate paper Zhang et al. [2017]. V a l i d a t i o n e r r o r ( t o p - 1 ) ( % ) 4 4 3 8 . . 9 0 5 b a a d v - s o f t s y n 1 0 0 1 2 0 1 4 0 1 6 0 a d s v e - * s - o s o f t s f t y s n y n ## References 1. Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project Adam: Building an efficient and scalable deep learning training system. OSDI'14, pages 571-582, 2014. 2. Adam Coates, Brody Huval, Tao Wang, David Wu, Bryan Catan- zaro, and Ng Andrew. Deep learning with cots hpc systems. In Proceedings of the 30th ICML, pages 1337-1345, 2013. 3. Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, MarcAurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In NIPS, 2012. 4. Olga Russakovsky et al. ImageNet Large Scale Visual Recognition Challenge. IJCV, pages 1-42, 2015. 5. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature em- bedding. arXiv preprint arXiv:1408.5093, 2014. 6. Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Computer Science Department, Uni- versity of Toronto, Tech. Rep, 1(4):7, 2009. 7. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Ad- vances in neural information processing systems, pages 1097- 1105, 2012. 8. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444, 2015. 9. Wei Zhang, Suyog Gupta, Xiangru Lin, Ji Liu, and Fei Wang. Model accuracy and runtime tradeoff in distributed deep learning: A sys- tematic study and its theoretical underpinning. Knowledge and Information Systems, (In submission), 2017.