{"type":"ScholarlyArticle","title":"Towards Fully 8-bit Integer Inference for the Transformer Model","authors":[{"name":"Ye Lin","slug":"ye-lin","given":"Ye","family":"Lin","affiliation":"Natural Language Processing Lab. , Northeastern University , Shenyang , China"},{"name":"Yanyang Li","slug":"yanyang-li","given":"Yanyang","family":"Li","affiliation":"Natural Language Processing Lab. , Northeastern University , Shenyang , China"},{"name":"Tengbo Liu","slug":"tengbo-liu","given":"Tengbo","family":"Liu","affiliation":"Natural Language Processing Lab. , Northeastern University , Shenyang , China"},{"name":"Tong Xiao","slug":"tong-xiao","given":"Tong","family":"Xiao","affiliation":"Natural Language Processing Lab. , Northeastern University , Shenyang , China; NiuTrans Research , Shenyang , China"},{"name":"Tongran Liu","slug":"tongran-liu","given":"Tongran","family":"Liu","affiliation":"CAS Key Laboratory of Behavioral Science , Institute of Psychology , CAS , Beijing , China"},{"name":"Jingbo Zhu","slug":"jingbo-zhu","given":"Jingbo","family":"Zhu","affiliation":"Natural Language Processing Lab. , Northeastern University , Shenyang , China; NiuTrans Research , Shenyang , China"}],"abstract":"8-bit integer inference, as a promising direction in reducing both the latency and storage of deep neural networks, has made great progress recently. On the other hand, previous systems still rely on 32-bit floating point for certain functions in complex models (e.g., Softmax in Transformer), and make heavy use of quantization and de-quantization. In this work, we show that after a principled modification on the Transformer architecture, dubbed Integer Transformer, an (almost) fully 8-bit integer inference algorithm Scale Propagation could be derived. De-quantization is adopted when necessary, which makes the network more efficient. Our experiments on WMT16 En<->Ro, WMT14 En<->De and En->Fr translation tasks as well as the WikiText-103 language modelling task show that the fully 8-bit Transformer system achieves comparable performance with the floating point baseline but requires nearly 4x less memory footprint.","doi":"10.24963/ijcai.2020/520","url":"https://ijcai.org/proceedings/2020/520","pdf":"https://ijcai.org/proceedings/2020/0520.pdf","markdown":"https://ijcai.org/proceedings/2020/520.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/520","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":520,"pages":{"first":3759,"last":3765},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Machine Translation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25279/file_25279.m3u8","length_sec":264,"provider":"hls"}],"references":[{"position":1,"raw":"[Baevski and Auli, 2019] Alexei Baevski and Michael Auli. Adaptive input representations for neural language model- ing. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.","title":"Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling","authors":"Auli Baevski","year":"2019","venue":"7th International Conference on Learning Representations, ICLR 2019","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhandare et al., 2019] Aishwarya Bhandare, Vamsi Sri- pathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. CoRR, abs/1906.00532, 2019.","title":"Efficient 8-bit quantization of transformer neural machine language translation model","authors":"Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, Vikram Saletore","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gong et al., 2018] Jiong Gong, Haihao Shen, Guoming Zhang, Xiaoli Liu, Shane Li, Ge Jin, Niharika Mahesh- wari, Evarist Fomenko, and Eden Segal. Highly effi- cient 8-bit low precision inference of convolutional neu- ral networks with intelcaffe. In Proceedings of the 1st on Reproducible Quality-Efficient Systems Tourna- ment on Co-designing Pareto-efficient Deep Learning, Re- QuEST@ASPLOS 2018, Williamsburg, VA, USA, March 24, 2018, page 2, 2018.","title":"Highly efficient 8-bit low precision inference of convolutional neural networks with intelcaffe","authors":"Jiong Gong, Haihao Shen, Guoming Zhang, Xiaoli Liu, Shane Li, Ge Jin, Niharika Maheshwari, Evarist Fomenko, Eden Segal","year":"2018","venue":"Proceedings of the 1st on Reproducible Quality-Efficient Systems Tournament on Co-designing Pareto-efficient Deep Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vi- sion, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"2015 IEEE International Conference on Computer Vision, ICCV 2015","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hoffer et al., 2018] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pages 2164-2174, 2018.","title":"Norm matters: efficient and accurate normalization schemes in deep networks","authors":"Elad Hoffer, Ron Banner, Itay Golan, Daniel Soudry","year":"2018","venue":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In Advances in Neural Informa- tion Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4107-4115, 2016.","title":"Binarized neural networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jacob et al., 2018] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2704-2713, 2018.","title":"Quantization and training of neural networks for efficient integer-arithmetic-only inference","authors":"Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G Howard, Hartwig Adam, Dmitry Kalenichenko","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson, 2018] Jeff Johnson. Rethinking floating point for deep learning. CoRR, abs/1811.01721, 2018.","title":"Rethinking floating point for deep learning","authors":"Jeff Johnson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2020] Yanyang Li, Qiang Wang, Tong Xiao, Ton- gran Liu, and Jingbo Zhu. Neural machine translation with joint representation. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.","title":"Neural machine translation with joint representation","authors":"Yanyang Li, Qiang Wang, Tong Xiao, Tongran Liu, Jingbo Zhu","year":"2020","venue":"Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Micikevicius et al., 2018] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings, 2018.","title":"Mixed precision training","authors":"Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu","year":"2018","venue":"6th International Conference on Learning Representations, ICLR 2018","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Prato et al., 2019] Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for im- proved translation. CoRR, abs/1910.10485, 2019.","title":"Fully quantized transformer for improved translation","authors":"Gabriele Prato, Ella Charlaix, Mehdi Rezagholizadeh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Quinn and Ballesteros, 2018] Jerry Quinn and Miguel Ballesteros. Pieces of eight: 8-bit neural machine transla- tion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 3 (Industry Papers), pages 114-120, 2018.","title":"Pieces of eight: 8-bit neural machine translation","authors":"Quinn, Ballesteros, Jerry Quinn, Miguel Ballesteros","year":"2018","venue":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Strubell et al., 2019] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Con- ference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Vol- ume 1: Long Papers, pages 3645-3650, 2019.","title":"Energy and policy considerations for deep learning in NLP","authors":"Emma Strubell, Ananya Ganesh, Andrew Mccallum","year":"2019","venue":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sze et al., 2017] Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. Efficient processing of deep neu- ral networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2295-2329, 2017.","title":"Efficient processing of deep neural networks: A tutorial and survey","authors":"Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, Joel S Emer","year":"2017","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2019] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Associ- ation for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1810-1822. Association for Computational Linguis- tics, 2019.","title":"Learning deep transformer models for machine translation","authors":"Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, Lidia S Chao","year":"2019","venue":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu, 2020] Ephrem Wu. Learning accurate integer transformer machine-translation models. CoRR, abs/2001.00926, 2020.","title":"Learning accurate integer transformer machine-translation models","authors":"Ephrem Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiao et al., 2019] Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Arti- ficial Intelligence, IJCAI 2019, Macao, China, August 10- 16, 2019, pages 5292-5298. ijcai.org, 2019.","title":"Sharing attention weights for fast transformer","authors":"Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, Tongran Liu","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019","doi":null,"url":null,"internal_url":"/proceedings/2019/735"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}