{"type":"ScholarlyArticle","title":"Cooperative Pruning in Cross-Domain Deep Neural Network Compression","authors":[{"name":"Shangyu Chen","slug":"shangyu-chen","given":"Shangyu","family":"Chen","affiliation":"Nanyang Technological University , Singapore"},{"name":"Wenya Wang","slug":"wenya-wang","given":"Wenya","family":"Wang","affiliation":"Nanyang Technological University , Singapore"},{"name":"Sinno Jialin Pan","slug":"sinno-jialin-pan","given":"Sinno Jialin","family":"Pan","affiliation":"Nanyang Technological University , Singapore"}],"abstract":"The advancement of deep models poses great challenges to real-world deployment because of the limited computational ability and storage space on edge devices. To solve this problem, existing works have made progress to compress deep models by pruning or quantization. However, most existing methods rely on a large amount of training data and a pre-trained model in the same domain. When only limited in-domain training data is available, these methods fail to perform well. This prompts the idea of transferring knowledge from a resource-rich source domain to a target domain with limited data to perform model compression.\r\nIn this paper, we propose a  method to perform cross-domain pruning by cooperatively training in both domains: taking advantage of data and a pre-trained model from the source domain to assist pruning in the target domain. Specifically, source and target pruned models are trained simultaneously and interactively, with source information transferred through the construction of a cooperative pruning mask. Our method significantly improves pruning quality in the target domain, and shed light to model compression in the cross-domain setting.","doi":"10.24963/ijcai.2019/291","url":"https://ijcai.org/proceedings/2019/291","pdf":"https://ijcai.org/proceedings/2019/0291.pdf","markdown":"https://ijcai.org/proceedings/2019/291.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/291","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":291,"pages":{"first":2102,"last":2108},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Bucilua et al., 2006] Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international con- ference on Knowledge discovery and data mining, pages 535-541. ACM, 2006.","title":"Model compression","authors":"Cristian Bucilua, Rich Caruana, Alexandru Niculescu-Mizil","year":"2006","venue":"Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2017] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer- wise optimal brain surgeon. In I.","title":"Learning to prune deep neural networks via layerwise optimal brain surgeon","authors":"Xin Dong, Shangyu Chen, Sinno Pan","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems 30, pages 4860-4874.","title":"Advances in Neural Information Processing Systems","authors":"U V Guyon, S Luxburg, H Bengio, R Wallach","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zis- serman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganin and Lempitsky, 2014] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backprop- agation. arXiv preprint arXiv:1409.7495, 2014.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor Lempitsky","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Griffin et al., 2007] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.","title":"Caltech-256 object category dataset","authors":"Gregory Griffin, Alex Holub, Pietro Perona","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2016] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Ad- vances In Neural Information Processing Systems, pages 1379-1387, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Yiwen Guo, Anbang Yao, Yurong Chen","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gupta et al., 2016] Saurabh Gupta, Judy Hoffman, and Ji- tendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 2827-2836, 2016.","title":"Cross modal distillation for supervision transfer","authors":"Saurabh Gupta, Judy Hoffman, Jitendra Malik","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Informa- tion Processing Systems, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hassibi and Stork, 1993] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information process- ing systems, pages 164-171, 1993.","title":"Second order derivatives for network pruning: Optimal brain surgeon","authors":"Babak Hassibi, David G Stork","year":"1993","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[LeCun et al., 1990] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touret- zky, editor, Advances in Neural Information Processing Systems 2, pages 598-605.","title":"Optimal brain damage","authors":"Yann Lecun, John S Denker, Sara A Solla","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowl- edge and data engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE Transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pan et al., 2011] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Net- works, 22(2):199-210, 2011.","title":"Domain adaptation via transfer component analysis","authors":"Ivor W Sinno Jialin Pan, James T Tsang, Qiang Kwok, Yang","year":"2011","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Romero et al., 2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tzeng et al., 2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confu- sion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.","title":"Deep domain confusion: Maximizing for domain invariance","authors":"Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, Trevor Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 7167-7176, 2017.","title":"Adversarial discriminative domain adaptation","authors":"Eric Tzeng, Judy Hoffman, Kate Saenko, Trevor Darrell","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}