{"type":"ScholarlyArticle","title":"AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search","authors":[{"name":"Daoyuan Chen","slug":"daoyuan-chen","given":"Daoyuan","family":"Chen","affiliation":"Alibaba Group"},{"name":"Yaliang Li","slug":"yaliang-li","given":"Yaliang","family":"Li","affiliation":"Alibaba Group"},{"name":"Minghui Qiu","slug":"minghui-qiu","given":"Minghui","family":"Qiu","affiliation":"Alibaba Group"},{"name":"Zhen Wang","slug":"zhen-wang","given":"Zhen","family":"Wang","affiliation":"Alibaba Group"},{"name":"Bofang Li","slug":"bofang-li","given":"Bofang","family":"Li","affiliation":"Alibaba Group"},{"name":"Bolin Ding","slug":"bolin-ding","given":"Bolin","family":"Ding","affiliation":"Alibaba Group"},{"name":"Hongbo Deng","slug":"hongbo-deng","given":"Hongbo","family":"Deng","affiliation":"Alibaba Group"},{"name":"Jun Huang","slug":"jun-huang","given":"Jun","family":"Huang","affiliation":"Alibaba Group"},{"name":"Wei Lin","slug":"wei-lin","given":"Wei","family":"Lin","affiliation":"Alibaba Group"},{"name":"Jingren Zhou","slug":"jingren-zhou","given":"Jingren","family":"Zhou","affiliation":"Alibaba Group"}],"abstract":"Large pre-trained language models such as BERT have shown their effectiveness in various natural language processing tasks. However, the huge parameter size makes them difficult to be deployed in real-time applications that require quick inference with limited resources. Existing methods compress BERT into small models while such compression is task-independent, i.e., the same compressed BERT for all different downstream tasks. Motivated by the necessity and benefits of task-oriented BERT compression, we propose a novel compression method, AdaBERT, that leverages differentiable Neural Architecture Search to automatically compress BERT into task-adaptive small models for specific tasks. We incorporate a task-oriented knowledge distillation loss to provide search hints and an efficiency-aware loss as search constraints, which enables a good trade-off between efficiency and effectiveness for task-adaptive BERT compression. We evaluate AdaBERT on several NLP tasks, and the results demonstrate that those task-adaptive compressed models are 12.7x to 29.3x faster than BERT in inference time and 11.5x to 17.0x smaller in terms of parameter size, while comparable performance is maintained.","doi":"10.24963/ijcai.2020/341","url":"https://ijcai.org/proceedings/2020/341","pdf":"https://ijcai.org/proceedings/2020/0341.pdf","markdown":"https://ijcai.org/proceedings/2020/341.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/341","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":341,"pages":{"first":2463,"last":2469},"keywords":["Natural Language Processing: Natural Language Processing","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24616/file_24616.m3u8","length_sec":281,"provider":"hls"}],"references":[{"position":1,"raw":"[Bai et al., 2018] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.","title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling","authors":"Bai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clark et al., 2019] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341, 2019.","title":"What does bert look at? an analysis of bert's attention","authors":"Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D Manning","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jang et al., 2017] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiao et al., 2019] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understand- ing. arXiv preprint arXiv:1909.10351, 2019.","title":"Tinybert: Distilling bert for natural language understanding","authors":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lan et al., 2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR, 2020.","title":"ALBERT: A lite BERT for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2019a] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. DARTS: Differentiable architecture search. In ICLR, 2019.","title":"DARTS: Differentiable architecture search","authors":"Liu","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2019b] Nelson F. Liu, Matt Gardner, Yonatan Be- linkov, Matthew E. Peters, and Noah A. Smith. Linguis- tic knowledge and transferability of contextual representa- tions. In NAACL, pages 1073-1094, 2019.","title":"Linguistic knowledge and transferability of contextual representations","authors":"Liu","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2019c] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2019] Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. In NeurIPS, pages 2229-2239, 2019.","title":"A tensorized transformer for language modeling","authors":"Ma","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maddison et al., 2017] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A con- tinuous relaxation of discrete random variables. In ICLR, 2017.","title":"The concrete distribution: A continuous relaxation of discrete random variables","authors":"Chris J Maddison, Andriy Mnih, Yee Whye Teh","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Michel et al., 2019] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NeurIPS, pages 14014-14024, 2019.","title":"Are sixteen heads really better than one? In NeurIPS","authors":"Paul Michel, Omer Levy, Graham Neubig","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Peters et al., 2018] Matthew E Peters, Mark Neumann, Mo- hit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word repre- sentations. In NAACL, 2018.","title":"Deep contextualized word representations","authors":"Mark Matthew E Peters, Mohit Neumann, Matt Iyyer, Christopher Gardner, Kenton Clark, Luke Lee, Zettlemoyer","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pham et al., 2018] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, pages 4092-4101, 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, Jeff Dean","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sanh et al., 2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled ver- sion of bert: smaller, faster, cheaper and lighter. In NeurIPS EMC 2 Workshop, 2019.","title":"Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter","authors":"Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf","year":"2019","venue":"NeurIPS EMC 2 Workshop","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shen et al., 2018] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self- attention for fast and memory-efficient sequence model- ing. In ICLR, 2018.","title":"Bi-directional block selfattention for fast and memory-efficient sequence modeling","authors":"Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shen et al., 2019] Sheng Shen, Zhen Dong, Jiayu Ye, Lin- jian Ma, Zhewei Yao, Amir Gholami, Michael W Ma- honey, and Kurt Keutzer. Q-bert: Hessian based ul- tra low precision quantization of bert. arXiv preprint arXiv:1909.05840, 2019.","title":"Q-bert: Hessian based ultra low precision quantization of bert","authors":"Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, Kurt Keutzer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2019] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model com- pression. In EMNLP, 2019.","title":"Patient knowledge distillation for bert model compression","authors":"Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang et al., 2019] Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136, 2019.","title":"Distilling taskspecific knowledge from bert into simple neural networks","authors":"Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, Jimmy Lin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tenney et al., 2019] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In ACL, pages 4593-4601, 2019.","title":"BERT rediscovers the classical NLP pipeline","authors":"Ian Tenney, Dipanjan Das, Ellie Pavlick","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019a] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.","title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","authors":"Wang","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019b] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint arXiv:1910.04732, 2019.","title":"Structured pruning of large language models","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2019] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, pages 10734-10742, 2019.","title":"Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search","authors":"Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, Kurt Keutzer","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2019] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In ICLR, 2019.","title":"SNAS: stochastic neural architecture search","authors":"Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zafrir et al., 2019] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In NeurIPS EMC 2 Workshop, 2019.","title":"Q8bert: Quantized 8bit bert","authors":"Ofir Zafrir, Guy Boudoukh, Peter Izsak, Moshe Wasserblat","year":"2019","venue":"NeurIPS EMC 2 Workshop","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2019] Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. Extreme language model com- pression with optimal subwords and shared projections. arXiv preprint arXiv:1909.11687, 2019.","title":"Extreme language model compression with optimal subwords and shared projections","authors":"Sanqiang Zhao, Raghav Gupta, Yang Song, Denny Zhou","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. ICLR, 2017.","title":"Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}