A Compression-Compilation Framework for On-mobile Real-time BERT Applications

Niu, Wei; Kong, Zhenglun; Yuan, Geng; Jiang, Weiwen; Guan, Jiexiong; Ding, Caiwen; Zhao, Pu; Liu, Sijia; Ren, Bin; Wang, Yanzhi

doi:10.24963/ijcai.2021/712

A Compression-Compilation Framework for On-mobile Real-time BERT Applications

Wei Niu, Zhenglun Kong, Geng Yuan, Weiwen Jiang, Jiexiong Guan, Caiwen Ding, Pu Zhao, Sijia Liu, Bin Ren, Yanzhi Wang

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Demo Track. Pages 5000-5003.

https://doi.org/10.24963/ijcai.2021/712

Keywords

Natural Language Processing: General
Knowledge Representation and Reasoning: General

Abstract

Transformer-based deep learning models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. In this paper, we propose a compression-compilation co-design framework that can guarantee the identified model meets both resource and real-time specifications of mobile devices. Our framework applies a compiler-aware neural architecture optimization method (CANAO), which can generate the optimal compressed model that balances both accuracy and latency. We are able to achieve up to 7.8x speedup compared with TensorFlow-Lite with only minor accuracy loss. We present two types of BERT applications on mobile devices: Question Answering (QA) and Text Generation. Both can be executed in real-time with latency as low as 45ms. Videos for demonstrating the framework can be found on https://www.youtube.com/watch?v=_WIRvK_2PZI

Wei Niu, Zhenglun Kong, Geng Yuan, Weiwen Jiang, Jiexiong Guan, Caiwen Ding, Pu Zhao, Sijia Liu, Bin Ren, and Yanzhi Wang. 2021. A Compression-Compilation Framework for On-mobile Real-time BERT Applications. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 5000–5003. https://doi.org/10.24963/ijcai.2021/712

@inproceedings{ijcai2021p712,
  title     = {A Compression-Compilation Framework for On-mobile Real-time BERT Applications},
  author    = {Niu, Wei and Kong, Zhenglun and Yuan, Geng and Jiang, Weiwen and Guan, Jiexiong and Ding, Caiwen and Zhao, Pu and Liu, Sijia and Ren, Bin and Wang, Yanzhi},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {5000--5003},
  year      = {2021},
  month     = {8},
  note      = {Demo Track},
  doi       = {10.24963/ijcai.2021/712},
  url       = {https://doi.org/10.24963/ijcai.2021/712},
}

TY  - CPAPER
TI  - A Compression-Compilation Framework for On-mobile Real-time BERT Applications
AU  - Niu, Wei
AU  - Kong, Zhenglun
AU  - Yuan, Geng
AU  - Jiang, Weiwen
AU  - Guan, Jiexiong
AU  - Ding, Caiwen
AU  - Zhao, Pu
AU  - Liu, Sijia
AU  - Ren, Bin
AU  - Wang, Yanzhi
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 5000
EP  - 5003
DO  - 10.24963/ijcai.2021/712
UR  - https://doi.org/10.24963/ijcai.2021/712
ER  -