{"type":"ScholarlyArticle","title":"FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer","authors":[{"name":"Yang Lin","slug":"yang-lin","given":"Yang","family":"Lin","affiliation":"MEGVII Technology"},{"name":"Tianyu Zhang","slug":"tianyu-zhang","given":"Tianyu","family":"Zhang","affiliation":"MEGVII Technology"},{"name":"Peiqin Sun","slug":"peiqin-sun","given":"Peiqin","family":"Sun","affiliation":""},{"name":"Zheng Li","slug":"zheng-li","given":"Zheng","family":"Li","affiliation":"MEGVII Technology"},{"name":"Shuchang Zhou","slug":"shuchang-zhou","given":"Shuchang","family":"Zhou","affiliation":"MEGVII Technology"}],"abstract":"Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.","doi":"10.24963/ijcai.2022/164","url":"https://ijcai.org/proceedings/2022/164","pdf":"https://ijcai.org/proceedings/2022/0164.pdf","markdown":"https://ijcai.org/proceedings/2022/164.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/164","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":164,"pages":{"first":1173,"last":1179},"keywords":["Computer Vision: Applications","Computer Vision: Machine Learning for Vision"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986619","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984762","length_sec":350,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2018] Jingyong Cai, Masashi Takemoto, and Hironori Nakajo. A deep look into logarithmic quantization of model pa- rameters in neural networks. In Proceedings of the 10th Interna- tional Conference on Advances in Information Technology, 2018.","title":"A deep look into logarithmic quantization of model parameters in neural networks","authors":"Cai","year":"2018","venue":"Proceedings of the 10th International Conference on Advances in Information Technology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choukroun et al., 2019] Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In IEEE/CVF International Conference on Computer Vision Workshops. IEEE, 2019.","title":"Low-bit quantization of neural networks for efficient inference","authors":"Yoni Choukroun, Eli Kravchik, Fan Yang, Pavel Kisilev","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Graham et al., 2021] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.01136, 2021.","title":"Levit: a vision transformer in convnet's clothing for faster inference","authors":"Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, Matthijs Douze","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2016] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Con- ference on Learning Representations, 2016.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. In International Conference on Machine Learning. PMLR, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jacob et al., 2018] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Quantization and training of neural networks for efficient integer-arithmetic-only inference","authors":"Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiao et al., 2020] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP, 2020.","title":"Tinybert: Distilling BERT for natural language understanding","authors":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu","year":"2020","venue":"Findings of the Association for Computational Linguistics: EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2021] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International Conference on Machine Learning. PMLR, 2021.","title":"I-bert: Integer-only bert quantization","authors":"Kim","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Ge- offrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019] Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for ob- ject detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.","title":"Fully quantized network for object detection","authors":"Li","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lian et al., 2019] Xiaocong Lian, Zhenyu Liu, Zhourui Song, Jiwu Dai, Wei Zhou, and Xiangyang Ji. High-performance fpga-based cnn accelerator with block-floating-point arithmetic. IEEE Trans- actions on Very Large Scale Integration (VLSI) Systems, 2019.","title":"High-performance fpga-based cnn accelerator with block-floating-point arithmetic","authors":"Xiaocong Lian, Zhenyu Liu, Zhourui Song, Jiwu Dai, Wei Zhou, Xiangyang Ji","year":"2019","venue":"IEEE Transactions on Very Large Scale Integration (VLSI) Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in con- text. In European Conference on Computer Vision. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2021a] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans- former: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021b] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. In Thirty-Fifth Conference on Neural Infor- mation Processing Systems, 2021.","title":"Post-training quantization for vision transformer","authors":"Liu","year":"2021","venue":"Thirty-Fifth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nagel et al., 2020] Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning. PMLR, 2020.","title":"Up or down? adaptive rounding for post-training quantization","authors":"Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, Tijmen Blankevoort","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rao et al., 2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Ji- wen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. arXiv preprint arXiv:2106.02034, 2021.","title":"Dynamicvit: Efficient vision transformers with dynamic token sparsification","authors":"Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, Cho-Jui Hsieh","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2020] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial In- telligence, 2020.","title":"Q-bert: Hessian based ultra low precision quantization of bert","authors":"Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, Kurt Keutzer","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distilla- tion through attention. In International Conference on Machine Learning. PMLR, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantiza- tion via bit-split and stitching. In International Conference on Machine Learning. PMLR, 2020.","title":"Towards accurate post-training network quantization via bit-split and stitching","authors":"Peisong Wang, Qiang Chen, Xiangyu He, Jian Cheng","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2021] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. arXiv preprint arXiv:2108.01390, 2021.","title":"Evo-vit: Slow-fast token evolution for dynamic vision transformer","authors":"Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, Xing Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic seg- mentation from a sequence-to-sequence perspective with trans- formers. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2021.","title":"Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers","authors":"Zheng","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2016] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gra- dients. arXiv preprint arXiv:1606.06160, 2016.","title":"Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients","authors":"Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}