{"type":"ScholarlyArticle","title":"Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error","authors":[{"name":"Taiji Suzuki","slug":"taiji-suzuki","given":"Taiji","family":"Suzuki","affiliation":"The University of Tokyo , Japan ,; Center for Advanced Intelligence Project , RIKEN , Japan ,"},{"name":"Hiroshi Abe","slug":"hiroshi-abe","given":"Hiroshi","family":"Abe","affiliation":"iPride Co., Ltd. , Japan ,"},{"name":"Tomoya Murata","slug":"tomoya-murata","given":"Tomoya","family":"Murata","affiliation":"NTT DATA Mathematical Systems Inc. , Japan ,"},{"name":"Shingo Horiuchi","slug":"shingo-horiuchi","given":"Shingo","family":"Horiuchi","affiliation":"NTT Data Corporation , Japan ,"},{"name":"Kotaro Ito","slug":"kotaro-ito","given":"Kotaro","family":"Ito","affiliation":"NTT DATA Mathematical Systems Inc. , Japan ,"},{"name":"Tokuma Wachi","slug":"tokuma-wachi","given":"Tokuma","family":"Wachi","affiliation":"NTT DATA Mathematical Systems Inc. , Japan ,"},{"name":"So Hirai","slug":"so-hirai","given":"So","family":"Hirai","affiliation":"NTT Data Corporation , Japan ,"},{"name":"Masatoshi Yukishima","slug":"masatoshi-yukishima","given":"Masatoshi","family":"Yukishima","affiliation":"NTT DATA Mathematical Systems Inc. , Japan ,"},{"name":"Tomoaki Nishimura","slug":"tomoaki-nishimura","given":"Tomoaki","family":"Nishimura","affiliation":"NTT Data Corporation , Japan ,"}],"abstract":"Compression techniques for deep neural network models are becoming very important\r\nfor the efficient execution of high-performance deep learning systems on edge-computing devices.\r\nThe concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based error bound.\r\nHowever, there is still huge gap between a practically effective compression method and its rigorous background of statistical learning theory. To resolve this issue, we develop a new theoretical framework for model compression\r\nand propose a new pruning method called {\\it spectral pruning} based on this framework.\r\nWe define the ``degrees of freedom'' to quantify the intrinsic dimensionality of a model\r\nby using the eigenvalue distribution of the covariance matrix across the internal nodes\r\nand show that the compression ability is essentially controlled by this quantity.\r\nMoreover, we present a sharp generalization error bound of the compressed model\r\nand characterize the bias--variance tradeoff induced by the compression procedure.\r\nWe apply our method to several datasets to justify our theoretical analyses and show the superiority of the the proposed method.","doi":"10.24963/ijcai.2020/393","url":"https://ijcai.org/proceedings/2020/393","pdf":"https://ijcai.org/proceedings/2020/0393.pdf","markdown":"https://ijcai.org/proceedings/2020/393.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/393","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":393,"pages":{"first":2839,"last":2846},"keywords":["Machine Learning: Deep Learning","Machine Learning: Deep-learning Theory","Machine Learning: Feature Selection; Learning Sparse Models","Machine Learning: Learning Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25510/file_25510.m3u8","length_sec":273,"provider":"hls"}],"references":[{"position":1,"raw":"[Aghasi et al., 2017] Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg. Net-trim: Convex pruning of deep neural networks with performance guarantee. In Advances in Neural Information Processing Systems 30, pages 3180-3189. 2017.","title":"Net-trim: Convex pruning of deep neural networks with performance guarantee","authors":"Alireza Aghasi, Afshin Abdi, Nam Nguyen, Justin Romberg","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2018] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Proceedings of the 35th International Conference on Machine Learn- ing, volume 80, pages 254-263. PMLR, 2018.","title":"Stronger generalization bounds for deep nets via a compression approach","authors":"Sanjeev Arora, Rong Ge, Behnam Neyshabur, Yi Zhang","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bach, 2017] Francis Bach. On the equivalence between ker- nel quadrature rules and random feature expansions. Jour- nal of Machine Learning Research, 18(21):1-38, 2017.","title":"On the equivalence between kernel quadrature rules and random feature expansions","authors":"Francis Bach","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bartlett et al., 2019] Peter Bartlett, Nick Harvey, Christo- pher Liaw, and Abbas Mehrabian. Nearly-tight VC- dimension and pseudodimension bounds for piecewise lin- ear neural networks. Journal of Machine Learning Re- search, 20(63):1-17, 2019.","title":"Nearly-tight VCdimension and pseudodimension bounds for piecewise linear neural networks","authors":"Peter Bartlett, Nick Harvey, Christopher Liaw, Abbas Mehrabian","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caponnetto and de Vito, 2007] Andrea Caponnetto and Ernesto de Vito. Optimal rates for regularized least- squares algorithm. Foundations of Computational Mathematics, 7(3):331-368, 2007.","title":"Andrea Caponnetto and Ernesto de Vito. Optimal rates for regularized leastsquares algorithm","authors":"De Caponnetto, Vito","year":"2007","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pat- tern Recognition, 2009, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Denil et al., 2013] Misha Denil, Babak Shakibi, Laurent Dinh, and Nando de Freitas. Predicting parameters in deep learning. In Advances in neural information processing systems 26, pages 2148-2156, 2013.","title":"Predicting parameters in deep learning","authors":"Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Denton et al., 2014] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems 27, pages 1269-1277. 2014.","title":"Exploiting linear structure within convolutional networks for efficient evaluation","authors":"Wojciech Emily L Denton, Joan Zaremba, Yann Bruna, Rob Lecun, Fergus","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dziugaite and Roy, 2017] Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, 2017.","title":"Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data","authors":"Karolina Gintare, Daniel M Dziugaite, Roy","year":"2017","venue":"Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gunasekar et al., 2018] Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems 31, pages 9482- 9491, 2018.","title":"Implicit bias of gradient descent on linear convolutional networks","authors":"Suriya Gunasekar, Jason D Lee, Daniel Soudry, Nati Srebro","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hardt et al., 2016] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochas- tic gradient descent. In Proceedings of the 33rd Interna- tional Conference on Machine Learning, volume 48, pages 1225-1234. PMLR, 2016.","title":"Train faster, generalize better: Stability of stochastic gradient descent","authors":"Moritz Hardt, Ben Recht, Yoram Singer","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 1389-1397, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krause and Golovin, 2014] Andreas Krause and Daniel Golovin. Submodular function maximization. In Tractability, 2014.","title":"Submodular function maximization","authors":"Golovin Krause, Andreas Krause, Daniel Golovin","year":"2014","venue":"Tractability","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krogh and Hertz, 1992] Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Ad- vances in Neural Information Processing Systems 5, pages 950-957, 1992.","title":"A simple weight decay can improve generalization","authors":"Hertz Krogh, Anders Krogh, John A Hertz","year":"1992","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lebedev and Lempitsky, 2016] Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, pages 2554-2564, 2016.","title":"Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage","authors":"Lempitsky Lebedev","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: a filter level pruning method for deep neural network compression. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pages 5068-5076, 2017.","title":"ThiNet: a filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Maas et al., 2013] Andrew Maas, Awni Hannun, and An- drew Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.","title":"Rectifier nonlinearities improve neural network acoustic models","authors":"Andrew Maas, Awni Hannun, Andrew Ng","year":"2013","venue":"ICML Workshop on Deep Learning for Audio, Speech and Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mallows, 1973] Colin L Mallows. Some comments on Cp. Technometrics, 15(4):661-675, 1973.","title":"Some comments on Cp","authors":"Colin L Mallows","year":"1973","venue":"Technometrics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Suzuki et al., 2020a] Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization error. arXiv preprint arXiv:1808.08558v2, 2020.","title":"Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization error","authors":"Suzuki","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Suzuki et al., 2020b] Taiji Suzuki, Hiroshi Abe, and To- moaki Nishimura. Compression based bound for non- compressed network: Unified generalization error analy- sis of large compressible deep neural network. In Interna- tional Conference on Learning Representations, 2020.","title":"Compression based bound for noncompressed network: Unified generalization error analysis of large compressible deep neural network","authors":"Suzuki","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Suzuki, 2018] Taiji Suzuki. Fast generalization error bound of deep learning from a kernel perspective. In Proceed- ings of the 21st International Conference on Artificial In- telligence and Statistics, volume 84, pages 1397-1406.","title":"Fast generalization error bound of deep learning from a kernel perspective","authors":"Taiji Suzuki, Suzuki","year":"2018","venue":"Proceedings of the 21st International Conference on Artificial In-telligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wager et al., 2013] Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In Advances in Neural Information Processing Systems 26, pages 351-359. 2013.","title":"Dropout training as adaptive regularization","authors":"Stefan Wager, Sida Wang, Percy S Liang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wan et al., 2013] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neu- ral networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning, vol- ume 28, pages 1058-1066. PMLR, 2013.","title":"Regularization of neural networks using dropconnect","authors":"Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, Rob Fergus","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems 29, pages 2074-2082. 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2018] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui- Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. NISP: Pruning net- works using neuron importance score propagation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194-9203, 2018.","title":"NISP: Pruning networks using neuron importance score propagation","authors":"Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, Larry S Davis","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2019] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz. Non-vacuous generalization bounds at the imagenet scale: a PAC- bayesian compression approach. In International Confer- ence on Learning Representations, 2019.","title":"Non-vacuous generalization bounds at the imagenet scale: a PACbayesian compression approach","authors":"Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, Peter Orbanz","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}