{"type":"ScholarlyArticle","title":"Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks","authors":[{"name":"Tuan Hoang","slug":"tuan-hoang","given":"Tuan","family":"Hoang","affiliation":"Singapore University of Technology and Design"},{"name":"Thanh-Toan Do","slug":"thanh-toan-do","given":"Thanh-Toan","family":"Do","affiliation":"University of Liverpool"},{"name":"Tam V. Nguyen","slug":"tam-v-nguyen","given":"Tam V.","family":"Nguyen","affiliation":"University of Dayton nguyenanhtuan"},{"name":"Ngai-Man Cheung","slug":"ngai-man-cheung","given":"Ngai-Man","family":"Cheung","affiliation":"Singapore University of Technology and Design"}],"abstract":"This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. However, this approach would result in some mismatch: the gradient descent updates full-precision weights, but it does not update the quantized weights. To address this issue, we propose a novel method that enables direct updating of quantized weights with learnable quantization levels to minimize the cost function using gradient descent. Second, to obtain low bit-width activations, existing works consider all channels equally. However, the activation quantizers could be biased toward a few channels with high-variance. To address this issue, we propose a method to take into account the quantization errors of individual channels. With this approach, we can learn activation quantizers that minimize the quantization errors in the majority of channels. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on the image classification task, using AlexNet, ResNet and MobileNetV2 architectures on CIFAR-100 and ImageNet datasets.","doi":"10.24963/ijcai.2020/292","url":"https://ijcai.org/proceedings/2020/292","pdf":"https://ijcai.org/proceedings/2020/0292.pdf","markdown":"https://ijcai.org/proceedings/2020/292.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/292","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":292,"pages":{"first":2111,"last":2118},"keywords":["Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25285/file_25285.m3u8","length_sec":273,"provider":"hls"}],"references":[{"position":1,"raw":"[Aojun et al., 2017] Zhou Aojun, Yao Anbang, Guo Yiwen, Xu Lin, and Chen Yurong. Incremental Network Quan- tization: Towards Lossless CNNs with Low-Precision Weights. In ICLR, 2017.","title":"Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights","authors":"Yao Zhou Aojun, Guo Anbang, Xu Yiwen, Chen Lin, Yurong","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2017] Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep Learning with Low Precision by Half-wave Gaussian Quantization. In CVPR, pages 5918- 5926, 2017.","title":"Deep Learning with Low Precision by Half-wave Gaussian Quantization","authors":"Cai","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training Deep Neural Networks with binary weights during propa- gations. In NIPS, pages 3123-3131. 2015.","title":"BinaryConnect: Training Deep Neural Networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, pages 248-255, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gong et al., 2019] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In ICCV, pages 4852-4861, 2019.","title":"Differentiable soft quantization: Bridging full-precision and low-bit neural networks","authors":"Gong","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu et al., 2019] Jiaxin Gu, Junhe Zhao, Xiaolong Jiang, Baochang Zhang, Liu Jianzhuang, Guodong Guo, and Rongrong Ji. Bayesian Optimized 1-bit CNNs. In ICCV, pages 4909-4917, 2019.","title":"Bayesian Optimized 1-bit CNNs","authors":"Jiaxin Gu, Junhe Zhao, Xiaolong Jiang, Baochang Zhang, Liu Jianzhuang, Guodong Guo, Rongrong Ji","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2015] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learn- ing with Limited Numerical Precision. In ICML, page 1737-1746, 2015.","title":"Deep Learning with Limited Numerical Precision","authors":"Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pages 770-778, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton, 2012] Geoffreye Hinton. Neural networks for ma- chine learning. Coursera, video lectures, 2012.","title":"Geoffreye Hinton. Neural networks for machine learning","authors":"Hinton","year":"2012","venue":"Coursera, video lectures","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hou and Kwok, 2018] Lu Hou and James T. Kwok. Loss- aware weight quantization of deep networks. In ICLR, 2018.","title":"Lossaware weight quantization of deep networks","authors":"Kwok ; ] Hou, Lu Hou, James T Kwok","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hubara et al., 2017] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. JMLR, 18(1):6869-6898, 2017.","title":"Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2017","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, page 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jung et al., 2019] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In CVPR, pages 4350-4359, 2019.","title":"Learning to quantize deep networks by optimizing quantization intervals with task loss","authors":"Jung","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Univer- sity of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li and Liu, 2017] Fengfu Li and Bin Liu. Ternary weight networks. In ICLR, 2017.","title":"Ternary weight networks","authors":"Li, Liu, Fengfu Li, Bin Liu","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2017] Zefan Li, Bingbing Ni, Wenjun Zhang, Xi- aokang Yang, and Wen Gao. Performance Guaranteed Network Acceleration via High-Order Residual Quantiza- tion. In ICCV, pages 2584-2592, 2017.","title":"Performance Guaranteed Network Acceleration via High-Order Residual Quantization","authors":"Zefan Li, Bingbing Ni, Wenjun Zhang, Xiaokang Yang, Wen Gao","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2016] Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. Fixed Point Quantization of Deep Convolutional Networks. In ICML, page 2849-2858, 2016.","title":"Fixed Point Quantization of Deep Convolutional Networks","authors":"Darryl D Lin, Sachin S Talathi, V Sreekanth Annapureddy","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2018] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-Real Net: Enhancing the Performance of 1-bit CNNs with Improved Representational Capability and Advanced Training Algo- rithm. In ECCV, pages 747-763, 2018.","title":"Bi-Real Net: Enhancing the Performance of 1-bit CNNs with Improved Representational Capability and Advanced Training Algorithm","authors":"Liu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2019] Chunlei Liu, Wenrui Ding, Xin Xia, Yuan Hu, Baochang Zhang, Jianzhuang Liu, Bohan Zhuang, and Guodong Guo. Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs. In IJCAI, pages 854-860, 2019.","title":"Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs","authors":"Chunlei Liu, Wenrui Ding, Xin Xia, Yuan Hu, Baochang Zhang, Jianzhuang Liu, Bohan Zhuang, Guodong Guo","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/120"},{"position":21,"raw":"[Miyashita et al., 2016] Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional Neural Net- works using Logarithmic Data Representation. CoRR, abs/1603.01025, 2016.","title":"Convolutional Neural Networks using Logarithmic Data Representation","authors":"Daisuke Miyashita, Edward H Lee, Boris Murmann","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Park et al., 2017] Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted-entropy-based quantization for deep neural networks. In CVPR, pages 5456-5464, 2017.","title":"Weighted-entropy-based quantization for deep neural networks","authors":"Park","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. XNOR-Net: Im- ageNet Classification Using Binary Convolutional Neural Networks. In ECCV, pages 525-542, 2016.","title":"XNOR-Net: Im-ageNet Classification Using Binary Convolutional Neural Networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In CVPR, pages 4510-4520, 2018.","title":"MobileNetV2: Inverted Residuals and Linear Bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2017] Wei Tang, Gang Hua, and Liang Wang. How to Train a Compact Binary Neural Network with High Accuracy? In AAAI, pages 2625-2631, 2017.","title":"How to Train a Compact Binary Neural Network with High Accuracy?","authors":"Wei Tang, Gang Hua, Liang Wang","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wan et al., 2018] Diwen Wan, Fumin Shen, Li Liu, Fan Zhu, Jie Qin, Ling Shao, and Heng Tao Shen. TBN: Con- volutional Neural Network with Ternary Inputs and Binary Weights. In ECCV, pages 322-339, 2018.","title":"TBN: Convolutional Neural Network with Ternary Inputs and Binary Weights","authors":"Wan","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks. In ECCV, pages 373-390, 2018.","title":"LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks","authors":"Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, Gang Hua","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2016] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. CoRR, abs/1606.06160, 2016.","title":"DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients","authors":"Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2018] Yiren Zhou, Seyed-Mohsen Moosavi- Dezfooli, Ngai-Man Cheung, and Pascal Frossard. Adap- tive quantization for deep neural network. In AAAI, pages 4596-4604, 2018.","title":"Adaptive quantization for deep neural network","authors":"Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, Pascal Frossard","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2017] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained Ternary Quantization. In ICLR, 2017.","title":"Trained Ternary Quantization","authors":"Zhu","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhuang et al., 2018] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian D. Reid. Towards Effective Low-Bitwidth Convolutional Neural Networks. In CVPR, pages 7920-7928, 2018.","title":"Towards Effective Low-Bitwidth Convolutional Neural Networks","authors":"Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, Ian D Reid","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhuang et al., 2019] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian D. Reid. Structured Binary Neural Networks for Accurate Image Classifi- cation and Semantic Segmentation. In CVPR, pages 413-422, 2019.","title":"Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation","authors":"Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, Ian D Reid","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}