{"type":"ScholarlyArticle","title":"Reducing Underflow in Mixed Precision Training by Gradient Scaling","authors":[{"name":"Ruizhe Zhao","slug":"ruizhe-zhao","given":"Ruizhe","family":"Zhao","affiliation":"Imperial College London; Networks , Inc"},{"name":"Brian Vogel","slug":"brian-vogel","given":"Brian","family":"Vogel","affiliation":"Preferred Networks , Inc."},{"name":"Tanvir Ahmed","slug":"tanvir-ahmed","given":"Tanvir","family":"Ahmed","affiliation":"Preferred Networks , Inc."},{"name":"Wayne Luk","slug":"wayne-luk","given":"Wayne","family":"Luk","affiliation":"Imperial College London"}],"abstract":"By leveraging the half-precision floating-point format (FP16) well supported by recent GPUs, mixed precision training (MPT) enables us to train larger models under the same or even smaller budget. However, due to the limited representation range of FP16, gradients can often experience severe underflow problems that hinder backpropagation and degrade model accuracy. MPT adopts loss scaling, which scales up the loss value just before backpropagation starts, to mitigate underflow by enlarging the magnitude of gradients. Unfortunately, scaling once is insufficient: gradients from distinct layers can each have different data distributions and require non-uniform scaling. Heuristics and hyperparameter tuning are needed to minimize these side-effects on loss scaling. We propose gradient scaling, a novel method that analytically calculates the appropriate scale for each gradient on-the-fly. It addresses underflow effectively without numerical problems like overflow and the need for tedious hyperparameter tuning. Experiments on a variety of networks and tasks show that gradient scaling can improve accuracy and reduce overall training effort compared with the state-of-the-art MPT.","doi":"10.24963/ijcai.2020/404","url":"https://ijcai.org/proceedings/2020/404","pdf":"https://ijcai.org/proceedings/2020/0404.pdf","markdown":"https://ijcai.org/proceedings/2020/404.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/404","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":404,"pages":{"first":2922,"last":2928},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26655/file_26655.m3u8","length_sec":854,"provider":"hls"}],"references":[{"position":1,"raw":"[Badrinarayanan et al., 2017] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A deep convo- lutional encoder-decoder architecture for image segmenta- tion. PAMI, 39(12):2481-2495, 2017.","title":"SegNet: A deep convolutional encoder-decoder architecture for image segmentation","authors":"Vijay Badrinarayanan, Alex Kendall, Roberto Cipolla","year":"2017","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blanchard et al., 2019] Pierre Blanchard, Nicholas J. Higham, Florent Lopez, Theo Mary, and Srikara Pranesh. Mixed Precision Block Fused Multiply-Add : Error Analysis and Application to GPU Tensor Cores. MIMS EPrint: 2019.18, 2019.","title":"Mixed Precision Block Fused Multiply-Add : Error Analysis and Application to GPU Tensor Cores","authors":"Pierre Blanchard, Nicholas J Higham, Florent Lopez, Theo Mary, Srikara Pranesh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choquette et al., 2018] Jack Choquette, Olivier Giroux, and Denis Foley. Volta: Performance and Programmability. IEEE Micro, 2018.","title":"Volta: Performance and Programmability","authors":"Choquette","year":"2018","venue":"IEEE Micro","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2017] Cheng-Yang Fu, Wei Liu, Ananth Ranga, Ambrish Tyagi, and Alexander C. Berg. DSSD : Decon- volutional Single Shot Detector. 2017.","title":"DSSD : Deconvolutional Single Shot Detector","authors":"Cheng-Yang Fu, Wei Liu, Ananth Ranga, Ambrish Tyagi, Alexander C Berg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feedfor- ward neural networks. In AISTATS, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia Kaiming, and He Facebook. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour","authors":"Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia Kaiming, He Facebook","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta, 2019] Geetika Gupta. Using Ten- sor Cores for Mixed-Precision Scientific Computing. https://devblogs.nvidia.com/ tensor-cores-mixed-precision-scientific-computing/, 2019. \"(accessed: 10.01.2019)\".","title":"Using Tensor Cores for Mixed-Precision Scientific Computing","authors":"Geetika Gupta","year":"2019","venue":null,"doi":null,"url":"https://devblogs.nvidia.com/tensor-cores-mixed-precision-scientific-computing/,2019","internal_url":null},{"position":8,"raw":"[Hanin, 2018] Boris Hanin. Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients? In NeurIPS, pages 582-591, 2018.","title":"Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?","authors":"Boris Hanin, Hanin","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pages 770-778, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Higham et al., 2019] Nicholas J Higham, Srikara Pranesh, and Mawussi Zounon. Squeezing a Matrix into Half Pre- cision, with an Application to Solving Linear Systems. SIAM J. Sci. Comput., 41(5):585-602, 2019.","title":"Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems","authors":"J Nicholas, Srikara Higham, Mawussi Pranesh, Zounon","year":"2019","venue":"SIAM J. Sci. Comput","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Higham, 1993] Nicholas J. Higham. The accuracy of float- ing point summation. SIAM J. Scientific Computing, 14(4):783-799, 1993.","title":"The accuracy of floating point summation","authors":"J Nicholas, Higham","year":"1993","venue":"SIAM J. Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hoffer et al., 2018] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accu- rate normalization schemes in deep networks. In NeurIPS, pages 2160-2170, 2018.","title":"Norm matters: efficient and accurate normalization schemes in deep networks","authors":"Elad Hoffer, Ron Banner, Itay Golan, Daniel Soudry","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, pages 448-456, 2015.","title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","authors":"Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.","title":"Adam: A Method for Stochastic Optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kuchaiev et al., 2018] Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Jason Li, Huyen Nguyen, Carl Case, and Paulius Micikevicius. Mixed- precision training for NLP and speech recognition with OpenSeq2Seq. arXiv preprint arXiv:1805.10387, 2018.","title":"Mixedprecision training for NLP and speech recognition with OpenSeq2Seq","authors":"Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Jason Vitaly Lavrukhin, Huyen Li, Carl Nguyen, Paulius Case, Micikevicius","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV, pages 21-37, 2016.","title":"SSD: Single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mellempudi et al., 2019] Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed Pre- cision Training With 8-bit Floating Point. arXiv preprint arXiv:1905.12334, 2019.","title":"Mixed Precision Training With 8-bit Floating Point","authors":"Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, Bharat Kaul","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Micikevicius et al., 2018] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In ICLR, 2018.","title":"Mixed precision training","authors":"Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Niitani et al., 2017] Yusuke Niitani, Toru Ogawa, Shunta Saito, and Masaki Saito. ChainerCV: a library for deep learning in computer vision. In ACM Multimedia, pages 1217-1220, 2017.","title":"ChainerCV: a library for deep learning in computer vision","authors":"Niitani","year":"2017","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Ima- geNet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Ima-geNet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sakr et al., 2019] Charbel Sakr, Naigang Wang, Chia-Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, and Kailash Gopalakrishnan. Accumulation bit-width scaling for ultra-low precision training of deep networks. In ICLR, 2019.","title":"Accumulation bit-width scaling for ultra-low precision training of deep networks","authors":"Charbel Sakr, Naigang Wang, Chia-Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, Kailash Gopalakrishnan","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Strubell et al., 2019] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In ACL, pages 3645-3650, 2019.","title":"Energy and policy considerations for deep learning in NLP","authors":"Emma Strubell, Ananya Ganesh, Andrew Mccallum","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tokui et al., 2019] Seiya Tokui, Ryosuke Okuta, Takuya Akiba, Yusuke Niitani, Toru Ogawa, Shunta Saito, Shuji Suzuki, Kota Uenishi, Brian Vogel, and Hiroyuki Ya- mazaki Vincent. Chainer: A deep learning framework for accelerating the research cycle. In KDD, pages 2002- 2011. ACM, 2019.","title":"Chainer: A deep learning framework for accelerating the research cycle","authors":"Tokui","year":"2019","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018] Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point num- bers. In NeurIPS, pages 7686-7695, 2018.","title":"Training deep neural networks with 8-bit floating point numbers","authors":"Wang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}