{"type":"ScholarlyArticle","title":"Soft Threshold Ternary Networks","authors":[{"name":"Weixiang Xu","slug":"weixiang-xu","given":"Weixiang","family":"Xu","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Xiangyu He","slug":"xiangyu-he","given":"Xiangyu","family":"He","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Tianli Zhao","slug":"tianli-zhao","given":"Tianli","family":"Zhao","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Qinghao Hu","slug":"qinghao-hu","given":"Qinghao","family":"Hu","affiliation":"Institute of Automation , Chinese Academy of Sciences; CAS Center for Excellence in Brain Science and Intelligence Technology"},{"name":"Peisong Wang","slug":"peisong-wang","given":"Peisong","family":"Wang","affiliation":"Institute of Automation , Chinese Academy of Sciences; CAS Center for Excellence in Brain Science and Intelligence Technology"},{"name":"Jian Cheng","slug":"jian-cheng","given":"Jian","family":"Cheng","affiliation":"Institute of Automation , Chinese Academy of Sciences; CAS Center for Excellence in Brain Science and Intelligence Technology"}],"abstract":"Large neural networks are difficult to deploy on mobile devices because of intensive computation and storage. To alleviate it, we study ternarization, a balance between efficiency and accuracy that quantizes both weights and activations into ternary values. In previous ternarized neural networks, a hard threshold Δ is introduced to determine quantization intervals. Although the selection of Δ greatly affects the training results, previous works estimate Δ via an approximation or treat it as a hyper-parameter, which is suboptimal. In this paper, we present the Soft Threshold Ternary Networks (STTN), which enables the model to automatically determine quantization intervals instead of depending on a hard threshold. Concretely, we replace the original ternary kernel with the addition of two binary kernels at training time, where ternary values are determined by the combination of two corresponding binary values. At inference time, we add up the two binary kernels to obtain a single ternary kernel. Our method dramatically outperforms current state-of-the-arts, lowering the performance gap between full-precision networks and extreme low bit networks. Experiments on ImageNet with AlexNet (Top-1 55.6%), ResNet-18 (Top-1 66.2%) achieves new state-of-the-art.","doi":"10.24963/ijcai.2020/318","url":"https://ijcai.org/proceedings/2020/318","pdf":"https://ijcai.org/proceedings/2020/0318.pdf","markdown":"https://ijcai.org/proceedings/2020/318.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/318","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":318,"pages":{"first":2298,"last":2304},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Machine Learning: Classification"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24369/file_24369.m3u8","length_sec":303,"provider":"hls"}],"references":[{"position":1,"raw":"[Cai et al., 2017] Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. pages 5406-5414, 2017.","title":"Deep learning with low precision by half-wave gaussian quantization","authors":"Cai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Choi, 2018] Jungwook Choi. Pact: Parameterized clipping activation for quantized neural networks. arXiv: Computer Vision and Pattern Recognition, 2018.","title":"Pact: Parameterized clipping activation for quantized neural networks","authors":"Jungwook Choi, Choi","year":"2018","venue":"arXiv: Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Courbariaux et al., 2014] Matthieu Courbariaux, Jean- Pierre David, and Yoshua Bengio. Low precision storage for deep learning. arXiv preprint arXiv:1412.7024, 2014.","title":"Low precision storage for deep learning","authors":"Matthieu Courbariaux, Jean-Pierre David, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jeanpierre David. Binaryconnect: training deep neural networks with binary weights during propaga- tions. pages 3123-3131, 2015.","title":"Binaryconnect: training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jeanpierre David","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Courbariaux et al., 2016] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Ben- gio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.","title":"Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1","authors":"Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Darabi et al., 2018] Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. Bn- n+: Improved binary network training. arXiv preprint arXiv:1812.11800, 2018.","title":"Bn-n+: Improved binary network training","authors":"Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, Vahid Partovi, Nia","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2015] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Con- ference on Machine Learning, pages 1737-1746, 2015.","title":"Deep learning with limited numerical precision","authors":"Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J Dal- ly. Deep compression: Compressing deep neural network- s with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2019] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 4340-4349, 2019.","title":"Filter pruning via geometric median for deep convolutional neural networks acceleration","authors":"Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, Yi Yang","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and J- eff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Howard et al., 2017] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, M Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: Computer Vision and Pattern Recog- nition, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, M Andreetto, Hartwig Adam","year":"2017","venue":"arXiv: Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural net- works with low rank expansions. arXiv preprint arX- iv:1405.3866, 2014.","title":"Speeding up convolutional neural networks with low rank expansions","authors":"Jaderberg","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2016] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:1605.04711, 2016.","title":"Ternary weight networks","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019] Yuhang Li, Xin Dong, Sai Qian Zhang, Haoli Bai, Yuanpeng Chen, and Wei Wang. Rtn: Reparameterized ternary network. arXiv preprint arX- iv:1912.02057, 2019.","title":"Rtn: Reparameterized ternary network","authors":"Yuhang Li, Xin Dong, Qian Sai, Haoli Zhang, Yuanpeng Bai, Wei Chen, Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2017] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pages 345-353, 2017.","title":"Towards accurate binary convolutional neural network","authors":"Lin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2018] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorith- m. In Proceedings of the European Conference on Com- puter Vision (ECCV), pages 722-737, 2018.","title":"Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm","authors":"Liu","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019] ChunLei Liu, Wenrui Ding, Xin Xia, Baochang Zhang, Jiaxin Gu, Jianzhuang Liu, Rongrong Ji, and Doermann David. Circulant binary convolutional networks: Enhancing the performance of 1-bit dcnns with circulant back propagation. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2019.","title":"Circulant binary convolutional networks: Enhancing the performance of 1-bit dcnns with circulant back propagation","authors":"Liu","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In European Conference on Computer Vision, pages 525-542. Springer, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tang et al., 2017] Wei Tang, Gang Hua, and Liang Wang. How to train a compact binary neural network with high accuracy? In Thirty-First AAAI Conference on Artificial Intelligence, 2017.","title":"How to train a compact binary neural network with high accuracy?","authors":"Wei Tang, Gang Hua, Liang Wang","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wan et al., 2018] Diwen Wan, Fumin Shen, Li Liu, Fan Zhu, Jie Qin, Ling Shao, and Heng Tao Shen. Tbn: Con- volutional neural network with ternary inputs and binary weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 315-332, 2018.","title":"Tbn: Convolutional neural network with ternary inputs and binary weights","authors":"Wan","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng. Two- step quantization for low-bit neural networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 4376-4384, 2018.","title":"Twostep quantization for low-bit neural networks","authors":"Wang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2016] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.","title":"Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients","authors":"Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2016] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:1612.01064, 2016.","title":"Trained ternary quantization","authors":"Zhu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhu et al., 2019] Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4923- 4932, 2019.","title":"Binary ensemble neural network: More bits per network or more networks per bit?","authors":"Shilin Zhu, Xin Dong, Hao Su","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zoph and Le, 2016] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.","title":"Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}