{"type":"ScholarlyArticle","title":"BiFSMN: Binary Neural Network for Keyword Spotting","authors":[{"name":"Haotong Qin","slug":"haotong-qin","given":"Haotong","family":"Qin","affiliation":"Beihang University; Bytedance AI Lab {qinhaotong ,"},{"name":"Xudong Ma","slug":"xudong-ma","given":"Xudong","family":"Ma","affiliation":"Beihang University"},{"name":"Yifu Ding","slug":"yifu-ding","given":"Yifu","family":"Ding","affiliation":"Beihang University"},{"name":"Xiaoyang Li","slug":"xiaoyang-li","given":"Xiaoyang","family":"Li","affiliation":"Bytedance AI Lab {qinhaotong ,"},{"name":"Yang Zhang","slug":"yang-zhang","given":"Yang","family":"Zhang","affiliation":"Bytedance AI Lab {qinhaotong ,"},{"name":"Yao Tian","slug":"yao-tian","given":"Yao","family":"Tian","affiliation":"Bytedance AI Lab {qinhaotong ,"},{"name":"Zejun Ma","slug":"zejun-ma","given":"Zejun","family":"Ma","affiliation":"Bytedance AI Lab {qinhaotong ,"},{"name":"Jie Luo","slug":"jie-luo","given":"Jie","family":"Luo","affiliation":"Beihang University"},{"name":"Xianglong Liu","slug":"xianglong-liu","given":"Xianglong","family":"Liu","affiliation":"Beihang University"}],"abstract":"The deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. However, computational resources for these networks are significantly constrained since they usually run on-call on edge devices. In this paper, we present BiFSMN, an accurate and extreme-efficient binary neural network for KWS. We first construct a High-frequency Enhancement Distillation scheme for the binarization-aware training, which emphasizes the high-frequency information from the full-precision network's representation that is more crucial for the optimization of the binarized network. Then, to allow the instant and adaptive accuracy-efficiency trade-offs at runtime, we also propose a Thinnable Binarization Architecture to further liberate the acceleration potential of the binarized network from the topology perspective. Moreover, we implement a Fast Bitwise Computation Kernel for BiFSMN on ARMv8 devices which fully utilizes registers and increases instruction throughput to push the limit of deployment efficiency. Extensive experiments show that BiFSMN outperforms existing binarization methods by convincing margins on various datasets and is even comparable with the full-precision counterpart (e.g., less than 3% drop on Speech Commands V1-12). We highlight that benefiting from the thinnable architecture and the optimized 1-bit implementation, BiFSMN can achieve an impressive 22.3x speedup and 15.5x storage-saving on real-world edge hardware.","doi":"10.24963/ijcai.2022/603","url":"https://ijcai.org/proceedings/2022/603","pdf":"https://ijcai.org/proceedings/2022/0603.pdf","markdown":"https://ijcai.org/proceedings/2022/603.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/603","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":603,"pages":{"first":4346,"last":4352},"keywords":["Natural Language Processing: Speech","Machine Learning: Convolutional Networks"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986543","length_sec":57,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984686","length_sec":411,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2014] Guoguo Chen, Carolina Parada, and Georg Heigold. Small-footprint keyword spotting using deep neural networks. In ICASSP, 2014.","title":"Small-footprint keyword spotting using deep neural networks","authors":"Chen","year":"2014","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Mengzhe Chen, Shiliang Zhang, Ming Lei, Yong Liu, Haitao Yao, and Jie Gao. Compact feed- forward sequential memory networks for small-footprint keyword spotting. In Interspeech, 2018.","title":"Compact feedforward sequential memory networks for small-footprint keyword spotting","authors":"Chen","year":"2018","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propaga- tions. In NeurIPS, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Courbariaux et al., 2016] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Ben- gio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv, 2016.","title":"Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1","authors":"Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ding et al., 2019] Ruizhou Ding, Ting-Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribu- tion for training binarized deep networks. In CVPR, 2019. [huawei noah, 2021] huawei noah. Bolt. https://github.com/ huawei-noah/bolt, 2021. Accessed: 2022-01-08.","title":"Regularizing activation distribution for training binarized deep networks","authors":"Ding","year":"2019","venue":"CVPR","doi":null,"url":"https://github.com/huawei-noah/bolt,2021","internal_url":null},{"position":6,"raw":"[Kim et al., 2021] Byeonggeun Kim, Simyung Chang, Jinkyu Lee, and Dooyong Sung. Broadcasted residual learning for efficient keyword spotting. arXiv, 2021.","title":"Broadcasted residual learning for efficient keyword spotting","authors":"Kim","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Liu et al., 2018] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with im- proved representational capability and advanced training algorithm. In ECCV, 2018.","title":"Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm","authors":"Liu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Martinez et al., 2020] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neu- ral networks with real-to-binary convolutions. In ICLR, 2020.","title":"Training binary neural networks with real-to-binary convolutions","authors":"Brais Martinez, Jing Yang, Adrian Bulat, Georgios Tzimiropoulos","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Meyer, 1992] Yves Meyer. Wavelets and Operators: Volume 1. Cambridge university press, 1992.","title":"Wavelets and Operators","authors":"Yves Meyer, Meyer","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Qin et al., 2020] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for ac- curate binary neural networks. In CVPR, 2020.","title":"Forward and backward information retention for accurate binary neural networks","authors":"Qin","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Qin et al., 2021] Haotong Qin, Zhongang Cai, Mingyuan Zhang, Yifu Ding, Haiyu Zhao, Shuai Yi, Xianglong Liu, and Hao Su. Bipointnet: Binary neural network for point clouds. In ICLR, 2021.","title":"Bipointnet: Binary neural network for point clouds","authors":"Qin","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qin et al., 2022] Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua YAN, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. BiBERT: Accurate fully binarized BERT. In ICLR, 2022.","title":"BiBERT: Accurate fully binarized BERT","authors":"Yifu Haotong Qin, Mingyuan Ding, Zhang, Yan Qinghua, Aishan Liu, Qingqing Dang, Ziwei Liu, Xianglong Liu","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In ECCV, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2020] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In ICLR, 2020.","title":"Fastspeech 2: Fast and high-quality end-to-end text to speech","authors":"Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rosso et al., 2001] Osvaldo A Rosso, Susana Blanco, Ju- liana Yordanova, Vasil Kolev, Alejandra Figliola, Martin Schürmann, and Erol Başar. Wavelet entropy: a new tool for analysis of short duration brain electrical signals. J. Neurosci. Methods, 2001.","title":"Wavelet entropy: a new tool for analysis of short duration brain electrical signals","authors":"Osvaldo A Rosso, Susana Blanco, Juliana Yordanova, Alejandra Vasil Kolev, Martin Figliola, Erol Schürmann, Başar","year":"2001","venue":"J. Neurosci. Methods","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sahu et al., 2021] Surya Kant Sahu, Sai Mitheran, Juhi Kamdar, and Meet Gandhi. Audiomer: A convo- lutional transformer for keyword spotting. ArXiv, abs/2109.10252, 2021.","title":"Audiomer: A convolutional transformer for keyword spotting","authors":"Sahu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shang et al., 2021] Hengchao Shang, Ting Hu, Daimeng Wei, Zongyao Li, Jianfei Feng, Zhengzhe Yu, Jiaxin Guo, Shaojun Li, Lizhi Lei, Shimin Tao, et al. Hw-tsc's partic- ipation in the wmt 2021 efficiency shared task. In Pro- ceedings of the Sixth Conference on Machine Translation, pages 781-786, 2021.","title":"Hw-tsc's participation in the wmt 2021 efficiency shared task","authors":"Hengchao Shang, Ting Hu, Daimeng Wei, Zongyao Li, Jianfei Feng, Zhengzhe Yu, Jiaxin Guo, Shaojun Li, Lizhi Lei, Shimin Tao","year":"2021","venue":"Proceedings of the Sixth Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tian et al., 2021] Yao Tian, Haitao Yao, Meng Cai, Yam- ing Liu, and Zejun Ma. Improving rnn transducer mod- eling for small-footprint keyword spotting. In ICASSP, 2021.","title":"Improving rnn transducer modeling for small-footprint keyword spotting","authors":"Haitao Yao Tian, Meng Yao, Yaming Cai, Zejun Liu, Ma","year":"2021","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Warden, 2018] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv:1804.03209, 2018.","title":"Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition","authors":"Warden","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2018] Xuerui Yang, Jiwei Li, and Xi Zhou. A novel pyramidal-fsmn architecture with lattice-free mmi for speech recognition. arXiv:1810.11352, 2018.","title":"A novel pyramidal-fsmn architecture with lattice-free mmi for speech recognition","authors":"Xuerui Yang, Jiwei Li, Xi Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2015] Shiliang Zhang, Cong Liu, Hui Jiang, Si Wei, Lirong Dai, and Yu Hu. Feedforward sequential memory networks: A new structure to learn long-term de- pendency. arXiv:1512.08301, 2015.","title":"Feedforward sequential memory networks: A new structure to learn long-term dependency","authors":"Shiliang Zhang, Cong Liu, Hui Jiang, Si Wei, Lirong Dai, Yu Hu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2017] Yundong Zhang, Naveen Suda, Liangzhen Lai, and Vikas Chandra. Hello edge: Keyword spotting on microcontrollers. arXiv:1711.07128, 2017.","title":"Hello edge: Keyword spotting on microcontrollers","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Shiliang Zhang, Ming Lei, Zhijie Yan, and Lirong Dai. Deep-fsmn for large vocabulary continu- ous speech recognition. In ICASSP, 2018.","title":"Deep-fsmn for large vocabulary continuous speech recognition","authors":"Shiliang Zhang, Ming Lei, Zhijie Yan, Lirong Dai","year":"2018","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019] Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao, and Tao Mei. dabnn: A super fast inference framework for binary neural networks on ARM devices. In ACM MM, 2019.","title":"dabnn: A super fast inference framework for binary neural networks on ARM devices","authors":"Zhang","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2016] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv, abs/1606.06160, 2016.","title":"Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients","authors":"Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}