{"type":"ScholarlyArticle","title":"SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference","authors":[{"name":"Boren Hu","slug":"boren-hu","given":"Boren","family":"Hu","affiliation":"Zhejiang University"},{"name":"Yun Zhu","slug":"yun-zhu","given":"Yun","family":"Zhu","affiliation":"Zhejiang University"},{"name":"Jiacheng Li","slug":"jiacheng-li","given":"Jiacheng","family":"Li","affiliation":"Zhejiang University"},{"name":"Siliang Tang","slug":"siliang-tang","given":"Siliang","family":"Tang","affiliation":"Zhejiang University"}],"abstract":"Dynamic early exiting has been proven to improve the inference speed of the pre-trained language model like BERT. However, all samples must go through all consecutive layers before early exiting and more complex samples usually go through more layers, which still exists redundant computation. In this paper, we propose a novel dynamic early exiting combined with layer skipping for BERT inference named SmartBERT, which adds a skipping gate and an exiting operator into each layer of BERT. SmartBERT can adaptively skip some layers and adaptively choose whether to exit. Besides, we propose cross-layer contrastive learning and combine it into our training phases to boost the intermediate layers and classifiers which would be beneficial for early exiting. To keep the inconsistent usage of skipping gates between training and inference phases, we propose a hard weight mechanism during training phase. We conduct experiments on eight classification datasets of the GLUE benchmark. Experimental results show that SmartBERT achieves 2-3× computation reduction with minimal accuracy drops compared with BERT and our method outperforms previous methods in both efficiency and accuracy. Moreover, in some complex datasets, we prove that the early exiting based on entropy hardly works, and the skipping mechanism is essential for reducing computation.","doi":"10.24963/ijcai.2023/563","url":"https://ijcai.org/proceedings/2023/563","pdf":"https://ijcai.org/proceedings/2023/0563.pdf","markdown":"https://ijcai.org/proceedings/2023/563.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/563","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":563,"pages":{"first":5067,"last":5075},"keywords":["Natural Language Processing: NLP: Text classification","Natural Language Processing: NLP: Language models"],"videos":[],"references":[{"position":1,"raw":"[Bentivogli et al., 2009] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recog- nizing textual entailment challenge. In TAC, 2009.","title":"The fifth pascal recognizing textual entailment challenge","authors":"Luisa Bentivogli, Peter Clark, Ido Dagan, Danilo Giampiccolo","year":"2009","venue":"TAC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey E. Hinton. A simple frame- work for contrastive learning of visual representations. In Proc. of ICML, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E Hinton","year":"2020","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proc. of NAACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dolan and Brockett, 2005] Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential para- phrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.","title":"Automatically constructing a corpus of sentential paraphrases","authors":"Brockett Dolan, Bill Dolan, Chris Brockett","year":"2005","venue":"Third International Workshop on Paraphrasing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proc. of EMNLP, 2021.","title":"Simcse: Simple contrastive learning of sentence embeddings","authors":"Tianyu Gao, Xingcheng Yao, Danqi Chen","year":"2021","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gordon et al., 2020] Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT: studying the ef- fects of weight pruning on transfer learning. CoRR, 2020.","title":"Compressing BERT: studying the effects of weight pruning on transfer learning","authors":"Mitchell A Gordon, Kevin Duh, Nicholas Andrews","year":"2020","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proc. of ICCV, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. of CVPR, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross B Girshick","year":"2020","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jacob et al., 2018] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proc. of CVPR, 2018.","title":"Quantization and training of neural networks for efficient integer-arithmetic-only inference","authors":"Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko","year":"2018","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiao et al., 2019] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understand- ing. arXiv preprint arXiv:1909.10351, 2019.","title":"Tinybert: Distilling bert for natural language understanding","authors":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lan et al., 2019] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of lan- guage representations. arXiv preprint arXiv:1909.11942, 2019.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levesque et al., 2012] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.","title":"The winograd schema challenge","authors":"Levesque","year":"2012","venue":"Thirteenth international conference on the principles of knowledge representation and reasoning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for ef- ficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2020] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. Fastbert: a self- distilling bert with adaptive inference time. arXiv preprint arXiv:2004.02178, 2020.","title":"Fastbert: a selfdistilling bert with adaptive inference time","authors":"Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, Qi Ju","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. In Advances in neural information processing systems, pages 8735-8745, 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100,000+ ques- tions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.","title":"Squad: 100,000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sanh et al., 2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled ver- sion of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.","title":"Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter","authors":"Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schwartz et al., 2020] Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A Smith. The right tool for the job: Matching model and instance complexities. arXiv preprint arXiv:2004.07453, 2020.","title":"The right tool for the job: Matching model and instance complexities","authors":"Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, Noah A Smith","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts","year":"2013","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Srivastava et al., 2015] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.","title":"Highway networks","authors":"Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018a] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bow- man. Glue: A multi-task benchmark and analysis plat- form for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.","title":"Glue: A multi-task benchmark and analysis platform for natural language understanding","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018b] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learn- ing dynamic routing in convolutional networks. In Proc. of ECCV, 2018.","title":"Skipnet: Learning dynamic routing in convolutional networks","authors":"Wang","year":"2018","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Warstadt et al., 2019] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Compu- tational Linguistics, 2019.","title":"Neural network acceptability judgments","authors":"Alex Warstadt, Amanpreet Singh, Samuel R Bowman","year":"2019","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Williams et al., 2017] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge cor- pus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel R Bowman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wolf et al., 2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State- of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.","title":"Huggingface's transformers: Stateof-the-art natural language processing","authors":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xin et al., 2020] Ji Xin, Raphael Tang, Jaejun Lee, Yao- liang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004.12993, 2020.","title":"Deebert: Dynamic early exiting for accelerating bert inference","authors":"Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, Jimmy Lin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Proc. of NeurIPS, 2019.","title":"Xlnet: Generalized autoregressive pretraining for language understanding","authors":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, Quoc V Le","year":"2019","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[You et al., 2020] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph con- trastive learning with augmentations. Proc. of NeurIPS, 2020.","title":"Graph contrastive learning with augmentations","authors":"Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zafrir et al., 2019] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learn- ing and Cognitive Computing-NeurIPS Edition (EMC2- NIPS), 2019.","title":"Q8bert: Quantized 8bit bert","authors":"Ofir Zafrir, Guy Boudoukh, Peter Izsak, Moshe Wasserblat","year":"2019","venue":"Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2022] Yun Zhu, Jianhao Guo, Fei Wu, and Sil- iang Tang. Rosa: A robust self-aligned framework for node-node graph contrastive learning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 3795-3801. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.","title":"Rosa: A robust self-aligned framework for node-node graph contrastive learning","authors":"Yun Zhu, Jianhao Guo, Fei Wu, Siliang Tang","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22","doi":null,"url":null,"internal_url":"/proceedings/2022/527"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}