{"type":"ScholarlyArticle","title":"Network Approximation using Tensor Sketching","authors":[{"name":"Shiva Prasad Kasiviswanathan","slug":"shiva-prasad-kasiviswanathan","given":"Shiva Prasad","family":"Kasiviswanathan","affiliation":"Amazon AWS AI , USA"},{"name":"Nina Narodytska","slug":"nina-narodytska","given":"Nina","family":"Narodytska","affiliation":"VMware Research , USA; Samsung Research America."},{"name":"Hongxia Jin","slug":"hongxia-jin","given":"Hongxia","family":"Jin","affiliation":"Samsung Research America , USA"}],"abstract":"Deep neural networks are powerful learning models that achieve state-of-the-art performance on many computer vision, speech, and language processing tasks. In this paper, we study a fundamental question that arises when designing deep network architectures: Given a target network architecture can we design a `smaller' network architecture that 'approximates' the operation of the target network? The question is, in part, motivated by the challenge of parameter reduction (compression) in modern deep neural networks, as the ever increasing storage and memory requirements of these networks pose a problem in resource constrained environments.In this work, we focus on deep convolutional neural network architectures, and propose a novel randomized tensor sketching technique that we utilize to develop a unified framework for approximating the operation of both the convolutional and fully connected layers. By applying the sketching technique along different tensor dimensions, we design changes to the convolutional and fully connected layers that substantially reduce the number of effective parameters in a network. We show that the resulting smaller network can be trained directly, and has a classification accuracy that is comparable to the original network.","doi":"10.24963/ijcai.2018/321","url":"https://ijcai.org/proceedings/2018/321","pdf":"https://ijcai.org/proceedings/2018/0321.pdf","markdown":"https://ijcai.org/proceedings/2018/321.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/321","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":321,"pages":{"first":2319,"last":2325},"keywords":["Machine Learning: Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2015] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In ICML, 2015.","title":"Compressing neural networks with the hashing trick","authors":"Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, Yixin Chen","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing convolutional neural networks in the frequency domain. In KDD, 2016.","title":"Compressing convolutional neural networks in the frequency domain","authors":"Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q Weinberger, Yixin Chen","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2015] Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In ICCV, pages 2857-2865, 2015.","title":"An exploration of parameter redundancy in deep networks with circulant projections","authors":"Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, Shi-Fu Chang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chetlur et al., 2014] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. ArXiv, 2014.","title":"cudnn: Efficient primitives for deep learning","authors":"Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, Evan Shelhamer","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Denil et al., 2013] Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al. Predicting parameters in deep learning. In NIPS, pages 2148-2156, 2013.","title":"Predicting parameters in deep learning","authors":"Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Garipov et al., 2016] Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensoriza- tion: compressing convolutional and fc layers alike. ArXiv, 2016.","title":"Ultimate tensorization: compressing convolutional and fc layers alike","authors":"Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, Dmitry Vetrov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gong et al., 2014] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. ArXiv, 2014.","title":"Compressing deep convolutional networks using vector quantization","authors":"Gong","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. Book in prepara- tion for MIT Press, 2016.","title":"Deep learning. Book in preparation for","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2016] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In NIPS, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Yiwen Guo, Anbang Yao, Yurong Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2016] Song Han, Huizi Mao, and William J Dally. A deep neural network compression pipeline: Pruning, quantization, huffman encoding. In ICLR, 2016.","title":"A deep neural network compression pipeline: Pruning, quantization, huffman encoding","authors":"Han","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hubara et al., 2016a] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In Advances in neural information processing systems, pages 4107-4115, 2016.","title":"Binarized neural networks","authors":"Hubara","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hubara et al., 2016b] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations. arXiv preprint arXiv:1609.07061, 2016.","title":"Quantized neural networks: Training neural networks with low precision weights and activations","authors":"Hubara","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Johnson and Lindenstrauss, 1984] William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189- 206):1, 1984.","title":"Extensions of lipschitz mappings into a hilbert space","authors":"B William, Joram Johnson, Lindenstrauss","year":"1984","venue":"Contemporary mathematics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2016] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compres- sion of deep convolutional neural networks for fast and low power mobile applications. In ICLR, 2016.","title":"Compression of deep convolutional neural networks for fast and low power mobile applications","authors":"Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, Dongjun Shin","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2016] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:1605.04711, 2016.","title":"Ternary weight networks","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2014] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.","title":"Network in network","authors":"Lin","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sindhwani et al., 2015] Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar. Structured transforms for small-footprint deep learning. In NIPS, 2015.","title":"Structured transforms for small-footprint deep learning","authors":"Vikas Sindhwani, Tara Sainath, Sanjiv Kumar","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Soulié et al., 2015] Guillaume Soulié, Vincent Gripon, and Maëlys Robert. Compression of deep neural networks on the fly. ArXiv, 2015.","title":"Compression of deep neural networks on the fly","authors":"Guillaume Soulié, Maëlys Vincent Gripon, Robert","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tai et al., 2016] Cheng Tai, Tong Xiao, Xiaogang Wang, et al. Convolutional neural networks with low-rank regular- ization. In ICLR, 2016.","title":"Convolutional neural networks with low-rank regularization","authors":"Cheng Tai, Tong Xiao, Xiaogang Wang","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Umans, 1998] Christopher Umans. The minimum equiva- lent dnf problem and shortest implicants. In Foundations of Computer Science, 1998. Proceedings. 39th Annual Sympo- sium on, pages 556-563. IEEE, 1998.","title":"The minimum equivalent dnf problem and shortest implicants","authors":"Christopher Umans","year":"1998","venue":"Proceedings. 39th Annual Symposium on","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2016] Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. Cnnpack: Packing convo- lutional neural networks in the frequency domain. In NIPS, 2016.","title":"Cnnpack: Packing convolutional neural networks in the frequency domain","authors":"Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, Chao Xu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Woodruff, 2014] David P. Woodruff. Sketching as a tool for numerical linear algebra. FnT-TCS, 2014.","title":"Sketching as a tool for numerical linear algebra","authors":"P David, Woodruff","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2015] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. ArXiv, 2015.","title":"Quantized convolutional neural networks for mobile devices","authors":"Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, Jian Cheng","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2015] Zichao Yang, Marcin Moczulski, Misha Denil, Nando de Freitas, Alex Smola, Le Song, and Ziyu Wang. Deep fried convnets. In ICCV, 2015.","title":"Deep fried convnets","authors":"Yang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2016] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:1612.01064, 2016.","title":"Trained ternary quantization","authors":"Zhu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}