{"type":"ScholarlyArticle","title":"KCNN: Kernel-wise Quantization to Remarkably Decrease Multiplications in Convolutional Neural Network","authors":[{"name":"Linghua Zeng","slug":"linghua-zeng","given":"Linghua","family":"Zeng","affiliation":"CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application Systems , University of Science and Technology of China , China"},{"name":"Zhangcheng Wang","slug":"zhangcheng-wang","given":"Zhangcheng","family":"Wang","affiliation":"CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application Systems , University of Science and Technology of China , China"},{"name":"Xinmei Tian","slug":"xinmei-tian","given":"Xinmei","family":"Tian","affiliation":"CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application Systems , University of Science and Technology of China , China"}],"abstract":"Convolutional neural networks (CNNs) have demonstrated state-of-the-art performance in computer vision tasks. However, the high computational power demand of running devices of recent CNNs has hampered many of their applications.  Recently, many methods have quantized the floating-point weights and activations to fixed-points or binary values to convert fractional arithmetic to integer or bit-wise arithmetic. However, since the distributions of values in CNNs are extremely complex, fixed-points or binary values lead to numerical information loss and cause performance degradation. On the other hand, convolution is composed of multiplications and accumulation, but the implementation of multiplications in hardware is more costly comparing with accumulation. We can preserve the rich information of floating-point values on dedicated low power devices by considerably decreasing the multiplications. In this paper, we quantize the floating-point weights in each kernel separately to multiple bit planes to remarkably decrease multiplications. We obtain a closed-form solution via an aggressive Lloyd algorithm and the fine-tuning is adopted to optimize the bit planes. Furthermore, we propose dual normalization to solve the pathological curvature problem during fine-tuning. Our quantized networks show negligible performance loss compared to their floating-point counterparts.","doi":"10.24963/ijcai.2019/588","url":"https://ijcai.org/proceedings/2019/588","pdf":"https://ijcai.org/proceedings/2019/0588.pdf","markdown":"https://ijcai.org/proceedings/2019/588.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/588","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":588,"pages":{"first":4234,"last":4242},"keywords":["Machine Learning: Deep Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learn- ing on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propaga- tions. In Advances in neural information processing sys- tems, pages 3123-3131, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cutler, 1952] Cassius C Cutler. Differential quantization of communication signals, July 29 1952. US Patent 2,605,361.","title":"Differential quantization of communication signals","authors":"C Cassius, Cutler","year":"1952","venue":"US Patent","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[David et al., 2007] Jean Pierre David, Kassem Kalach, and Nicolas Tittley. Hardware complexity of modular multi- plication and exponentiation. IEEE Transactions on Com- puters, 56(10):1308-1319, 2007.","title":"Hardware complexity of modular multiplication and exponentiation","authors":"Jean Pierre, David, Kassem Kalach, Nicolas Tittley","year":"2007","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2017] Yinpeng Dong, Renkun Ni, Jianguo Li, Yurong Chen, Jun Zhu, and Hang Su. Learning accurate low-bit deep neural networks with stochastic quantization. arXiv preprint arXiv:1708.01001, 2017.","title":"Learning accurate low-bit deep neural networks with stochastic quantization","authors":"Dong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Figurnov et al., 2016] Mikhail Figurnov, Aizhan Ibraimova, Dmitry P Vetrov, and Pushmeet Kohli. Perforatedcnns: Acceleration through elimination of redundant convolu- tions. In Advances in neural information processing sys- tems, pages 947-955, 2016.","title":"Perforatedcnns: Acceleration through elimination of redundant convolutions","authors":"Mikhail Figurnov, Aizhan Ibraimova, P Dmitry, Pushmeet Vetrov, Kohli","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.","title":"Deep learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2015] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Con- ference on Machine Learning, pages 1737-1746, 2015.","title":"Deep learning with limited numerical precision","authors":"Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural informa- tion processing systems, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hinton et al., 2012] Geoffrey Hinton, Nitsh Srivastava, and Kevin Swersky. Neural networks for machine learning. Coursera, video lectures, 264, 2012.","title":"Neural networks for machine learning","authors":"Geoffrey Hinton, Nitsh Srivastava, Kevin Swersky","year":"2012","venue":"Coursera, video lectures","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In Advances in neural information processing systems, pages 4107-4115, 2016.","title":"Binarized neural networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ioannou et al., 2017] Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Criminisi. Deep roots: Im- proving cnn efficiency with hierarchical filter groups. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1231-1240, 2017.","title":"Deep roots: Improving cnn efficiency with hierarchical filter groups","authors":"Yani Ioannou, Duncan Robertson, Roberto Cipolla, Antonio Criminisi","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2015] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530, 2015.","title":"Compression of deep convolutional neural networks for fast and low power mobile applications","authors":"Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, Dongjun Shin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Leng et al., 2018] Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural network: Squeeze the last bit out with admm. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018.","title":"Extremely low bit neural network: Squeeze the last bit out with admm","authors":"Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, Rong Jin","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2016] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:1605.04711, 2016.","title":"Ternary weight networks","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2016] Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, pages 2849-2858, 2016.","title":"Fixed point quantization of deep convolutional networks","authors":"Darryl Lin, Sachin Talathi, Sreekanth Annapureddy","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2017] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pages 344-352, 2017.","title":"Towards accurate binary convolutional neural network","authors":"Xiaofan Lin, Cong Zhao, Wei Pan","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lloyd, 1982] Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129-137, 1982.","title":"Least squares quantization in pcm","authors":"Lloyd ;, Stuart Lloyd","year":"1982","venue":"IEEE transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In European Conference on Computer Vision, pages 525-542. Springer, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Salimans and Kingma, 2016] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameteriza- tion to accelerate training of deep neural networks. In Ad- vances in Neural Information Processing Systems, pages 901-909, 2016.","title":"Weight normalization: A simple reparameterization to accelerate training of deep neural networks","authors":"Kingma ; Salimans, Tim Salimans, P Diederik, Kingma","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tai et al., 2015] Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067, 2015.","title":"Convolutional neural networks with low-rank regularization","authors":"Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wolf, 2002] Wayne Wolf. Modern VLSI design: system-on- chip design. Pearson Education, 2002.","title":"Modern VLSI design: system-onchip design","authors":"Wayne Wolf","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2018] Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. arXiv preprint arXiv:1802.04680, 2018.","title":"Training and inference with integers in deep neural networks","authors":"Shuang Wu, Guoqi Li, Feng Chen, Luping Shi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2017] Tien-Ju Yang, Yu-Hsin Chen, and Vivi- enne Sze. Designing energy-efficient convolutional neu- ral networks using energy-aware pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5687-5695, 2017.","title":"Designing energy-efficient convolutional neural networks using energy-aware pruning","authors":"Tien-Ju Yang, Yu-Hsin Chen, Vivienne Sze","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2018] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui- Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. Nisp: Pruning net- works using neuron importance score propagation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194-9203, 2018.","title":"Nisp: Pruning networks using neuron importance score propagation","authors":"Yu","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zeng and Tian, 2018] Linghua Zeng and Xinmei Tian. Ac- celerating convolutional neural networks by removing in- terspatial and interkernel redundancies. IEEE transactions on cybernetics, 2018.","title":"Accelerating convolutional neural networks by removing interspatial and interkernel redundancies","authors":"Tian Zeng, Linghua Zeng, Xinmei Tian","year":"2018","venue":"IEEE transactions on cybernetics","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2018] Xiangyu Zhang, Xinyu Zhou, Mengx- iao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6848-6856, 2018.","title":"Shufflenet: An extremely efficient convolutional neural network for mobile devices","authors":"Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2016] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:1612.01064, 2016.","title":"Trained ternary quantization","authors":"Zhu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}