{"type":"ScholarlyArticle","title":"Progressive Blockwise Knowledge Distillation for Neural Network Acceleration","authors":[{"name":"Hui Wang","slug":"hui-wang","given":"Hui","family":"Wang","affiliation":"Zhejiang University , Hangzhou , China"},{"name":"Hanbin Zhao","slug":"hanbin-zhao","given":"Hanbin","family":"Zhao","affiliation":"Zhejiang University , Hangzhou , China"},{"name":"Xi Li","slug":"xi-li","given":"Xi","family":"Li","affiliation":"Zhejiang University , Hangzhou , China"},{"name":"Xu Tan","slug":"xu-tan","given":"Xu","family":"Tan","affiliation":"Lab , Huawei Technologies , Hangzhou , China"}],"abstract":"As an important and challenging problem in machine learning and computer vision, neural network acceleration essentially aims to enhance the computational efficiency without sacrificing the model accuracy too much. In this paper, we propose a progressive blockwise learning scheme for teacher-student model distillation at the subnetwork block level. The proposed scheme is able to distill the knowledge of the entire teacher network by locally extracting the knowledge of each block in terms of progressive blockwise function approximation. Furthermore, we propose a structure design criterion for the student subnetwork block, which is able to effectively preserve the original receptive field from the teacher network. Experimental results demonstrate the effectiveness of the proposed scheme against the state-of-the-art approaches.","doi":"10.24963/ijcai.2018/384","url":"https://ijcai.org/proceedings/2018/384","pdf":"https://ijcai.org/proceedings/2018/0384.pdf","markdown":"https://ijcai.org/proceedings/2018/384.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/384","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":384,"pages":{"first":2769,"last":2775},"keywords":["Computer Vision: Computer Vision","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Ba and Caruana, 2014] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS, pages 2654- 2662, 2014.","title":"Do deep nets really need to be deep?","authors":"Caruana Ba, Jimmy Ba, Rich Caruana","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bucila et al., 2006] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. 2006.","title":"Model compression","authors":"Cristian Bucila, Rich Caruana, Alexandru Niculescu-Mizil","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propaga- tions. In NIPS, pages 3123-3131, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In ICCV, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2014] Geoffrey Hinton, Oriol Vinyals, and J- eff Dean. Distilling the knowledge in a neural network. NIPS Workshop, 2014.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2014","venue":"NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2016] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neu- ron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250, 2016.","title":"Network trimming: A data-driven neuron pruning approach towards efficient deep architectures","authors":"Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In NIPS, pages 4107-4115, 2016.","title":"Binarized neural networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural net- works with low rank expansions. In BMVC, 2014.","title":"Speeding up convolutional neural networks with low rank expansions","authors":"Jaderberg","year":"2014","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolution- al architecture for fast feature embedding. In ACM MM, pages 675-678, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017a] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.","title":"Pruning filters for efficient convnets","authors":"Li","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017b] Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detec- tion. In CVPR, pages 7341-7349, 2017.","title":"Mimicking very efficient network for object detection","authors":"Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017c] Zefan Li, Bingbing Ni, Wenjun Zhang, Xi- aokang Yang, and Wen Gao. Performance guaranteed net- work acceleration via high-order residual quantization. In ICCV, 2017.","title":"Performance guaranteed network acceleration via high-order residual quantization","authors":"Li","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.","title":"Network in network","authors":"Lin","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2016] Shaohui Lin, Rongrong Ji, Xiaowei Guo, Xuelong Li, et al. Towards convolutional neural network- s compression via global error reconstruction. In IJCAI, pages 1753-1759, 2016.","title":"Towards convolutional neural networks compression via global error reconstruction","authors":"Lin","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/251"},{"position":18,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Molchanov et al., 2017] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolu- tional neural networks for resource efficient transfer learn- ing. In ICLR, 2017.","title":"Pruning convolutional neural networks for resource efficient transfer learning","authors":"Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nowak and Corso, 2018] Theodore S Nowak and Jason J Corso. Deep net triage: Assessing the criticality of net- work layers by structural compression. arXiv preprint arX- iv:1801.04651, 2018.","title":"Deep net triage: Assessing the criticality of network layers by structural compression","authors":"S Theodore, Jason J Nowak, Corso","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In ECCV, pages 525-542. Springer, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Romero et al., 2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sainath et al., 2013] Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhad- ran. Low-rank matrix factorization for deep neural net- work training with high-dimensional output targets. In I- CASSP, pages 6655-6659, 2013.","title":"Low-rank matrix factorization for deep neural network training with high-dimensional output targets","authors":"Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, Bhuvana Ramabhadran","year":"2013","venue":"I-CASSP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, pages 4278-4284, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tai et al., 2016] Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. In ICLR, 2016.","title":"Convolutional neural networks with low-rank regularization","authors":"Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2016] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolution- al neural networks for mobile devices. In CVPR, pages 4820-4828, 2016.","title":"Quantized convolutional neural networks for mobile devices","authors":"Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, Jian Cheng","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2017] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. In ICLR, 2017.","title":"Incremental network quantization: Towards lossless cnns with low-precision weights","authors":"Zhou","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}