{"type":"ScholarlyArticle","title":"Automatic Mixed-Precision Quantization Search of BERT","authors":[{"name":"Changsheng Zhao","slug":"changsheng-zhao","given":"Changsheng","family":"Zhao","affiliation":""},{"name":"Ting Hua","slug":"ting-hua","given":"Ting","family":"Hua","affiliation":""},{"name":"Yilin Shen","slug":"yilin-shen","given":"Yilin","family":"Shen","affiliation":""},{"name":"Qian Lou","slug":"qian-lou","given":"Qian","family":"Lou","affiliation":""},{"name":"Hongxia Jin","slug":"hongxia-jin","given":"Hongxia","family":"Jin","affiliation":""}],"abstract":"Pre-trained language models such as BERT have shown remarkable effectiveness in various natural language processing tasks. However, these models usually contain millions of parameters, which prevent them from the practical deployment on resource-constrained devices. Knowledge distillation, Weight pruning, and Quantization are known to be the main directions in  model compression. However, compact models obtained through knowledge distillation  may suffer from significant accuracy drop even for a relatively small compression ratio. On the other hand, there are only a few attempts based on quantization designed for natural language processing tasks, and they usually require manual setting on hyper-parameters. In this paper, we proposed an automatic mixed-precision quantization framework designed for BERT that can conduct quantization and pruning simultaneously. Specifically, our proposed method leverages Differentiable Neural Architecture Search to assign scale and precision for parameters in each sub-group automatically, and at the same pruning out redundant groups of parameters. Extensive evaluations on BERT downstream tasks reveal that our proposed method beats baselines by providing the same performance with much smaller model size.\r\nWe also show the possibility of obtaining the extremely light-weight model by combining our solution with orthogonal methods such as DistilBERT.","doi":"10.24963/ijcai.2021/472","url":"https://ijcai.org/proceedings/2021/472","pdf":"https://ijcai.org/proceedings/2021/0472.pdf","markdown":"https://ijcai.org/proceedings/2021/472.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/472","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Machine Learning","track":"Main Track","number":472,"pages":{"first":3427,"last":3433},"keywords":["Natural Language Processing: NLP Applications and Tools","Natural Language Processing: Text Classification","Machine Learning: Deep Learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}