{"type":"ScholarlyArticle","title":"RAPQ: Rescuing Accuracy for Power-of-Two Low-bit Post-training Quantization","authors":[{"name":"Hongyi Yao","slug":"hongyi-yao","given":"Hongyi","family":"Yao","affiliation":"Peking University"},{"name":"Pu Li","slug":"pu-li","given":"Pu","family":"Li","affiliation":"Peking University"},{"name":"Jian Cao","slug":"jian-cao","given":"Jian","family":"Cao","affiliation":"Peking University"},{"name":"Xiangcheng Liu","slug":"xiangcheng-liu","given":"Xiangcheng","family":"Liu","affiliation":"Peking University"},{"name":"Chenying Xie","slug":"chenying-xie","given":"Chenying","family":"Xie","affiliation":"Peking University"},{"name":"Bingzhang Wang","slug":"bingzhang-wang","given":"Bingzhang","family":"Wang","affiliation":"Peking University"}],"abstract":"We introduce a Power-of-Two post-training quantization( PTQ) method for deep neural network that meets hardware requirements and does not call for long-time retraining. PTQ requires a small set of calibration data and is easier for deployment, but results in lower accuracy than Quantization-Aware Training( QAT). Power-of-Two quantization can convert the multiplication introduced by quantization and dequantization to bit-shift that is adopted by many efficient accelerators. However, the Power-of-Two scale has fewer candidate values, which leads to more rounding or clipping errors. We propose a novel Power-of-Two PTQ framework, dubbed RAPQ, which dynamically adjusts the Power-of-Two scales of the whole network instead of statically determining them layer by layer. It can theoretically trade off the rounding error and clipping error of the whole network. Meanwhile, the reconstruction method in RAPQ is based on the BN information of every unit. Extensive experiments on ImageNet prove the excellent performance of our proposed method. Without bells and whistles, RAPQ can reach accuracy of 65% and 48% on ResNet-18 and MobileNetV2 respectively with  weight INT2 activation INT4. We are the first to propose PTQ for the more constrained but hardware-friendly Power-of-Two quantization and prove that it can achieve nearly the same accuracy as SOTA PTQ method. The code will be released.","doi":"10.24963/ijcai.2022/219","url":"https://ijcai.org/proceedings/2022/219","pdf":"https://ijcai.org/proceedings/2022/0219.pdf","markdown":"https://ijcai.org/proceedings/2022/219.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/219","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":219,"pages":{"first":1573,"last":1579},"keywords":["Computer Vision: Machine Learning for Vision","Multidisciplinary Topics and Applications: AI Hardware","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986502","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984645","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Banner et al., 2019] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances in Neural In- formation Processing Systems, 2019.","title":"Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment","authors":"Banner","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2020] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178, 2020.","title":"Zeroq: A novel zero shot quantization framework","authors":"Cai","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gong et al., 2019] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks, 2019.","title":"Differentiable soft quantization: Bridging full-precision and low-bit neural networks","authors":"Gong","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hopfield and Tank, 1985] John J Hopfield and David W Tank. \"neural\" computation of decisions in optimization problems. Biological cybernetics, 52(3):141-152, 1985.","title":"neural\" computation of decisions in optimization problems","authors":"J John, David W Hopfield, Tank","year":"1985","venue":"Biological cybernetics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hubara et al., 2020] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.10518, 2020.","title":"Improving post training neural quantization: Layer-wise calibration and integer programming","authors":"Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, Daniel Soudry","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jain et al., 2019] Sambhav R Jain, Albert Gural, Michael Wu, and Chris H Dick. Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks. arXiv preprint arXiv:1903.08066, 2019.","title":"Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks","authors":"Albert Sambhav R Jain, Michael Gural, Chris H Wu, Dick","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krishnamoorthi, 2018] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient in- ference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018.","title":"Quantizing deep convolutional networks for efficient inference: A whitepaper","authors":"Raghuraman Krishnamoorthi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[LeCun et al., 2012] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Effi- cient backprop. In Neural networks: Tricks of the trade, pages 9-48. Springer, 2012.","title":"Efficient backprop","authors":"Léon Yann A Lecun, Genevieve B Bottou, Klaus-Robert Orr, Müller","year":"2012","venue":"Neural networks: Tricks of the trade","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019] Yuhang Li, Xin Dong, and Wei Wang. Addi- tive powers-of-two quantization: An efficient non-uniform discretization for neural networks. In International Con- ference on Learning Representations, 2019.","title":"Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks","authors":"Yuhang Li, Xin Dong, Wei Wang","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2020] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantiza- tion by block reconstruction. In International Conference on Learning Representations, 2020.","title":"Brecq: Pushing the limit of post-training quantization by block reconstruction","authors":"Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, Shi Gu","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021] Yuhang Li, Mingzhu Shen, Jian Ma, Yan Ren, Mingxin Zhao, Qi Zhang, Ruihao Gong, Fengwei Yu, and Junjie Yan. Mqbench: Towards reproducible and deployable model quantization benchmark. In Thirty- fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 1), 2021.","title":"Mqbench: Towards reproducible and deployable model quantization benchmark","authors":"Yuhang Li, Mingzhu Shen, Jian Ma, Yan Ren, Mingxin Zhao, Qi Zhang, Ruihao Gong, Fengwei Yu, Junjie Yan","year":"2021","venue":"Thirtyfifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Louizos et al., 2018] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l 0 regularization. International Conference on Learning Representations (ICLR), 2018.","title":"Learning sparse neural networks through l 0 regularization","authors":"Christos Louizos, Max Welling, Diederik P Kingma","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mittal et al., 2011] Arpit Mittal, Andrew Zisserman, and Philip HS Torr. Hand detection using multiple proposals. In Bmvc, volume 2, page 5, 2011.","title":"Hand detection using multiple proposals","authors":"Arpit Mittal, Andrew Zisserman, Philip Hs Torr","year":"2011","venue":"Bmvc","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Miyashita et al., 2016] Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01025, 2016.","title":"Convolutional neural networks using logarithmic data representation","authors":"Daisuke Miyashita, Edward H Lee, Boris Murmann","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nagel et al., 2020] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quanti- zation. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.","title":"Up or down? adaptive rounding for post-training quantization","authors":"Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, Tijmen Blankevoort","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nagel et al., 2021] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021.","title":"A white paper on neural network quantization","authors":"Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, Tijmen Blankevoort","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radosavovic et al., 2020] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436, 2020.","title":"Designing network design spaces","authors":"Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Redmon and Farhadi, 2018] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.","title":"Yolov3: An incremental improvement","authors":"Joseph Redmon, Ali Farhadi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International journal of computer vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training net- work quantization via bit-split and stitching. In Proc. 37nd Int. Conf. Mach. Learn.(ICML), 2020.","title":"Towards accurate post-training network quantization via bit-split and stitching","authors":"Wang","year":"2020","venue":"Proc. 37nd Int. Conf. Mach. Learn.(ICML)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}