{"type":"ScholarlyArticle","title":"Learning Low-precision Neural Networks without Straight-Through Estimator (STE)","authors":[{"name":"Zhi-Gang Liu","slug":"zhi-gang-liu","given":"Zhi-Gang","family":"Liu","affiliation":"Arm Machine Learning Research Lab"},{"name":"Matthew Mattina","slug":"matthew-mattina","given":"Matthew","family":"Mattina","affiliation":"Arm Machine Learning Research Lab"}],"abstract":"The Straight-Through Estimator (STE) is widely used for back-propagating gradients through the quantization function, but the STE technique lacks a complete theoretical understanding. We propose an alternative methodology called alpha-blending (AB), which quantizes neural networks to low precision using stochastic gradient descent (SGD). Our AB method avoids STE approximation by replacing the quantized weight in the loss function by an affine combination of the quantized weight w_q and the corresponding full-precision weight w with non-trainable scalar coefficient alpha and (1- alpha). During training, alpha is gradually increased from 0 to 1; the gradient updates to the weights are through the full precision term, (1-alpha) * w, of the affine combination; the model is converted from full-precision to low precision progressively. To evaluate the AB method, a 1-bit BinaryNet on CIFAR10 dataset and 8-bits, 4-bits MobileNet v1, ResNet_50 v1/2 on ImageNet are trained using the alpha-blending approach, and the evaluation indicates that AB improves top-1 accuracy by 0.9\\%, 0.82\\% and 2.93\\% respectively compared to the results of STE based quantization.","doi":"10.24963/ijcai.2019/425","url":"https://ijcai.org/proceedings/2019/425","pdf":"https://ijcai.org/proceedings/2019/0425.pdf","markdown":"https://ijcai.org/proceedings/2019/425.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/425","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":425,"pages":{"first":3066,"last":3072},"keywords":["Machine Learning: Classification","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron C Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propaga- tions. CoRR, abs/1511.00363, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dally, 2015] William Dally. Nips tutorial 2015. https://media.nips.cc/Conferences/2015/tutorialslides/Dally- NIPS-Tutorial-2015.pdf, 2015.","title":"Nips tutorial","authors":"William Dally","year":"2015","venue":null,"doi":null,"url":"https://media.nips.cc/Conferences/2015/tutorialslides/Dally-NIPS-Tutorial-2015.pdf","internal_url":null},{"position":4,"raw":"[Goetschalckx et al., 2018] Koen Goetschalckx, Bert Moons, Patrick Wambacq, and Marian Verhelst. Effi- ciently combining svd, pruning, clustering and retraining for enhanced neural network compression. pages 1-6, 06 2018.","title":"Efficiently combining svd, pruning, clustering and retraining for enhanced neural network compression","authors":"Koen Goetschalckx, Bert Moons, Patrick Wambacq, Marian Verhelst","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural net- work with pruning, trained quantization and huffman cod- ing. CoRR, abs/1510.00149, 2015.","title":"Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2012] Geoffrey Hinton, li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Phuongtrang Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29:82-97, 11 2012.","title":"Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups","authors":"Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-Rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Phuongtrang Nguyen, Tara Sainath, Brian Kingsbury","year":"2012","venue":"Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton, 2012] G. Hinton. Neural networks for machine learning, 2012.","title":"Neural networks for machine learning","authors":"G Hinton","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Howard et al., 2017a] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. CoRR, abs/1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Howard","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Howard et al., 2017b] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. CoRR, abs/1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Howard","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hubara et al., 2016a] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4107- 4115.","title":"Binarized neural networks","authors":"Hubara","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hubara et al., 2016b] Itay Hubara, Daniel Soudry, and Ran El Yaniv. Binarized neural networks. CoRR, abs/1602.02505, 2016. Withdrawn.","title":"Binarized neural networks","authors":"Hubara","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Iandola et al., 2016] Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size. CoRR, abs/1602.07360, 2016.","title":"Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size","authors":"Forrest N Iandola, Matthew W Moskewicz, Khalid Ashraf, Song Han, William J Dally, Kurt Keutzer","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krishnamoorthi, 2018] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. CoRR, abs/1806.08342, 2018.","title":"Quantizing deep convolutional networks for efficient inference: A whitepaper","authors":"Raghuraman Krishnamoorthi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2017] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6):84- 90, May 2017.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2017","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Leng et al., 2017] Cong Leng, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural network: Squeeze the last bit out with ADMM. CoRR, abs/1707.09870, 2017.","title":"Extremely low bit neural network: Squeeze the last bit out with ADMM","authors":"Cong Leng, Hao Li, Shenghuo Zhu, Rong Jin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li and Liu, 2016] Fengfu Li and Bin Liu. Ternary weight networks. CoRR, abs/1605.04711, 2016.","title":"Ternary weight networks","authors":"Li, Liu, Fengfu Li, Bin Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Penghang Yin, 2018] Shuai Zhang Stanley Osher Yingyong Qi Jack Xin Penghang Yin, Jiancheng Lyu. Understanding straight-through estimator in training activation quantized neural nets. 2018.","title":"Understanding straight-through estimator in training activation quantized neural nets","authors":"Shuai Zhang, Stanley Osher, Yingyong Qi, Jack Xin, Penghang Yin, Jiancheng Lyu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rastegari et al., 2016a] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. CoRR, abs/1603.05279, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Rastegari","year":"2016","venue":"Mohammad Rastegari, Vicente Or","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rastegari et al., 2016b] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. CoRR, abs/1603.05279, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Rastegari","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rastegari et al., 2016c] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. CoRR, abs/1603.05279, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Rastegari","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xue et al., 2013] J Xue, J Li, and Y Gong. Restructuring of deep neural network acoustic models with singular value decomposition. Proceedings of the Annual Conference of the International Speech Communication Association, IN- TERSPEECH, pages 2365-2369, 01 2013.","title":"Restructuring of deep neural network acoustic models with singular value decomposition","authors":"Xue, Y Li, Gong","year":"2013","venue":"Proceedings of the Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yin et al., 2018] Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks. arXiv e-prints, page arXiv:1808.05240, Aug 2018.","title":"Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks","authors":"Yin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou et al., 2017] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quanti- zation: Towards lossless cnns with low-precision weights. CoRR, abs/1702.03044, 2017.","title":"Incremental network quantization: Towards lossless cnns with low-precision weights","authors":"Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}