{"type":"ScholarlyArticle","title":"Dynamically Hierarchy Revolution: DirNet for Compressing Recurrent Neural Network on Mobile Devices","authors":[{"name":"Jie Zhang","slug":"jie-zhang","given":"Jie","family":"Zhang","affiliation":"Arizona State University , Tempe , AZ , USA"},{"name":"Xiaolong Wang","slug":"xiaolong-wang","given":"Xiaolong","family":"Wang","affiliation":"Samsung Research America , Mountain View , CA , USA"},{"name":"Dawei Li","slug":"dawei-li","given":"Dawei","family":"Li","affiliation":"Samsung Research America , Mountain View , CA , USA"},{"name":"Yalin Wang","slug":"yalin-wang","given":"Yalin","family":"Wang","affiliation":"Arizona State University , Tempe , AZ , USA"}],"abstract":"Recurrent neural networks (RNNs) achieve cutting-edge performance on a variety of problems. However, due to their high computational and memory demands, deploying RNNs on resource constrained mobile devices is a challenging task. To guarantee minimum accuracy loss with higher compression rate and driven by the mobile resource requirement, we introduce a novel model compression approach DirNet based on an optimized fast dictionary learning algorithm, which 1) dynamically mines the dictionary atoms of the projection dictionary matrix within layer to adjust the compression rate 2) adaptively changes the sparsity of sparse codes cross the hierarchical layers. Experimental results on language model and an ASR model trained with a 1000h speech dataset demonstrate that our method significantly outperforms prior approaches. Evaluated on off-the-shelf mobile devices, we are able to reduce the size of original model by eight times with real-time model inference and negligible accuracy loss.","doi":"10.24963/ijcai.2018/429","url":"https://ijcai.org/proceedings/2018/429","pdf":"https://ijcai.org/proceedings/2018/0429.pdf","markdown":"https://ijcai.org/proceedings/2018/429.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/429","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":429,"pages":{"first":3089,"last":3096},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, et al. Ten- sorflow: A system for large-scale machine learning. In OSDI, pages 265-283, GA, 2016. USENIX Association.","title":"Tensorflow: A system for large-scale machine learning","authors":"Martín Abadi, Paul Barham","year":"2016","venue":"OSDI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Yuntao Chen, Naiyan Wang, and Zhaox- iang Zhang. Darkrank: Accelerating deep metric learn- ing via cross sample similarities transfer. arXiv preprint arXiv:1707.01220, 2017.","title":"Darkrank: Accelerating deep metric learning via cross sample similarities transfer","authors":"Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, et al. Learning phrase representations using RNN encoder- decoder for statistical machine translation. abs/1406.1078, 2014.","title":"Learning phrase representations using RNN encoderdecoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Combettes and Wajs, 2005] Patrick L Combettes and Valérie R Wajs. Signal recovery by proximal forward- backward splitting. Multiscale Modeling & Simulation, 4(4):1168-1200, 2005.","title":"Signal recovery by proximal forwardbackward splitting","authors":"L Patrick, Combettes, Valérie R Wajs","year":"2005","venue":"Multiscale Modeling & Simulation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Davis and Mermelstein, 1990] Steven B Davis and Paul Mermelstein. Comparison of parametric representations # neurons of each layer Params WER (G) WER (O) 500, 500, 500, 500, 500 10.3M 12.7 12.7 350, 375, 395, 405, 410 8.6M 12.3 12.3 270, 305, 335, 345, 350 7.2M 12.5 12.3 175, 215, 245, 260, 265 5.4M 12.5 12.4 120, 150, 180, 195, 200 4.1M 12.6 12.5 80, 105, 130, 145, 150 3.1M 12.9 12.5 50, 70, 90, 100, 110 2.3M 13.2 12.7 30, 45, 55, 65, 75 1.7M 14.4 12.9 25, 35, 45, 50, 55 1.3M 16.6","title":"Comparison of parametric representations # neurons of each layer Params WER","authors":"B Steven, Paul Davis, Mermelstein","year":null,"venue":"G) WER (O)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"9 Table 3: The Word error rates (%) (WER) on the testing set as vary- ing the dimensions of each projection matrices D l by adaptively Ad- justing the hierarchical structure of RNN. for monosyllabic word recognition in continuously spo- ken sentences. In Readings in speech recognition, pages 65-74. Elsevier, 1990.","title":"9 Table 3: The Word error rates (%) (WER) on the testing set as varying the dimensions of each projection matrices D l by adaptively Adjusting the hierarchical structure of RNN. for monosyllabic word recognition in continuously spoken sentences","authors":null,"year":"1990","venue":"Readings in speech recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Denil et al., 2013] Misha Denil, Babak Shakibi, et al. Pre- dicting parameters in deep learning. In NIPS, pages 2148- 2156, 2013.","title":"Predicting parameters in deep learning","authors":"Misha Denil, Babak Shakibi","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Friedman et al., 2007] Jerome Friedman, Trevor Hastie, et al. Pathwise coordinate optimization. The Annals of Applied Statistics, 1(2):302-332, 2007.","title":"Pathwise coordinate optimization","authors":"Jerome Friedman, Trevor Hastie","year":"2007","venue":"The Annals of Applied Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Geng et al., 2017] Yanyan Geng, Guohui Zhang, Weizhi Li, Yi Gu, Gaoyuan Liang, Jingbin Wang, Yanbin Wu, Nitin Patil, and Jing-Yan Wang. A novel image tag completion method based on convolutional neural network. In Inter- national Conference on Artificial Neural Networks, pages 539-546. Springer, 2017.","title":"A novel image tag completion method based on convolutional neural network","authors":"Yanyan Geng, Guohui Zhang, Weizhi Li, Yi Gu, Gaoyuan Liang, Jingbin Wang, Yanbin Wu, Nitin Patil, Jing-Yan Wang","year":"2017","venue":"International Conference on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Graves et al., 2006] Alex Graves, Santiago Fernández, et al. Connectionist temporal classification: labelling unseg- mented sequence data with recurrent neural networks. In ICML, pages 369-376. ACM, 2006.","title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","authors":"Alex Graves, Santiago Fernández","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, et al. Speech recognition with deep recurrent neural net- works. In ICASSP, pages 6645-6649. IEEE, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed","year":"2013","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han et al., 2015a] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Han et al., 2015b] Song Han, Jeff Pool, et al. Learning both weights and connections for efficient neural network. In NIPS, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Han et al., 2017] Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang, et al. Ese: Efficient speech recogni- tion engine with sparse lstm on fpga. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field- Programmable Gate Arrays, pages 75-84. ACM, 2017.","title":"Ese: Efficient speech recognition engine with sparse lstm on fpga","authors":"Han","year":"2017","venue":"Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hitziger et al., 2013] Sebastian Hitziger, Maureen Clerc, et al. Jitter-adaptive dictionary learning-application to multi-trial neuroelectric signals. arXiv preprint arXiv:1301.3611, 2013.","title":"Jitter-adaptive dictionary learning-application to multi-trial neuroelectric signals","authors":"Sebastian Hitziger, Maureen Clerc","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kingsbury, 2009] Brian Kingsbury. Lattice-based optimiza- tion of sequence classification criteria for neural-network acoustic modeling. In ICASSP, pages 3761-3764, 2009.","title":"Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling","authors":"Brian Kingsbury","year":"2009","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lei et al., 2013] Xin Lei, Andrew W Senior, et al. Accu- rate and compact large vocabulary speech recognition on mobile devices. In INTERSPEECH. Citeseer, 2013.","title":"Accurate and compact large vocabulary speech recognition on mobile devices","authors":"Xin Lei, Andrew W Senior","year":"2013","venue":"INTERSPEECH. Citeseer","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2017] Dawei Li, Xiaolong Wang, and Deguang Kong. Deeprebirth: Accelerating deep neural net- work execution on mobile devices. arXiv preprint arXiv:1708.04728, 2017.","title":"Deeprebirth: Accelerating deep neural network execution on mobile devices","authors":"Dawei Li, Xiaolong Wang, Deguang Kong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2014] Binbin Lin, Qingyang Li, et al. Stochastic coordinate coding and its application for drosophila gene expression pattern annotation. arXiv:1407.8147, 2014.","title":"Stochastic coordinate coding and its application for drosophila gene expression pattern annotation","authors":"Binbin Lin, Qingyang Li","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mairal et al., 2009] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In ICML, pages 689-696. ACM, 2009.","title":"Online dictionary learning for sparse coding","authors":"Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Marcus et al., 1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330, 1993.","title":"Building a large annotated corpus of english: The penn treebank","authors":"Mary Mitchell P Marcus, Ann Marcinkiewicz, Beatrice Santorini","year":"1993","venue":"Computational linguistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mikolov et al., 2011] Tomàš Mikolov, S. Kombrink, et al. Extensions of recurrent neural network language model. In ICASSP, pages 5528-5531, May 2011.","title":"Extensions of recurrent neural network language model","authors":"Mikolov","year":"2011","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Narang et al., 2017] Sharan Narang, Gregory Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. ICLR, 2017.","title":"Exploring sparsity in recurrent neural networks","authors":"Sharan Narang, Gregory Diamos, Shubho Sengupta, Erich Elsen","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Panayotov et al., 2015] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP, pages 5206-5210. IEEE, 2015.","title":"Librispeech: an asr corpus based on public domain audio books","authors":"Vassil Panayotov, Guoguo Chen, Daniel Povey, Sanjeev Khudanpur","year":"2015","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Prabhavalkar et al., 2016] Rohit Prabhavalkar, Ouais Al- sharif, et al. On the compression of recurrent neural net- works with an application to lvcsr acoustic modeling for embedded speech recognition. In ICASSP, pages 5970- 5974. IEEE, 2016.","title":"On the compression of recurrent neural networks with an application to lvcsr acoustic modeling for embedded speech recognition","authors":"Rohit Prabhavalkar, Ouais Alsharif","year":"2016","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sainath et al., 2013] Tara N Sainath, Brian Kingsbury, et al. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In ICASSP, pages 6655-6659, 2013.","title":"Low-rank matrix factorization for deep neural network training with high-dimensional output targets","authors":"Tara N Sainath, Brian Kingsbury","year":"2013","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sak et al., 2014] Hasim Sak, Andrew Senior, et al. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, 2014.","title":"Long short-term memory recurrent neural network architectures for large scale acoustic modeling","authors":"Hasim Sak, Andrew Senior","year":"2014","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sak et al., 2015] Has ¸im Sak, Andrew Senior, et al. Learning acoustic frame labeling for speech recognition with recur- rent neural networks. In ICASSP, pages 4280-4284. IEEE, 2015.","title":"Has ¸im Sak, Andrew Senior, et al. Learning acoustic frame labeling for speech recognition with recurrent neural networks","authors":"Sak","year":"2015","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267-288, 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani, Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society. Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tseng, 2001] Paul Tseng. Convergence of a block co- ordinate descent method for nondifferentiable minimiza- tion. Journal of optimization theory and applications, 109(3):475-494, 2001.","title":"Convergence of a block coordinate descent method for nondifferentiable minimization","authors":"Paul Tseng","year":"2001","venue":"Journal of optimization theory and applications","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Vinyals et al., 2017] Oriol Vinyals, Alexander Toshev, et al. Show and tell: Lessons learned from the 2015 mscoco im- age captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652-663, 2017.","title":"Show and tell: Lessons learned from the 2015 mscoco image captioning challenge","authors":"Vinyals","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2018] Chen Xu, Jianqiang Yao, et al. Alternating multi-bit quantization for recurrent neural networks. In ICLR, 2018.","title":"Alternating multi-bit quantization for recurrent neural networks","authors":"Chen Xu, Jianqiang Yao","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xue et al., 2013] Jian Xue, Jinyu Li, and Yifan Gong. Re- structuring of deep neural network acoustic models with singular value decomposition. In INTERSPEECH, pages 2365-2369, 2013.","title":"Restructuring of deep neural network acoustic models with singular value decomposition","authors":"Jian Xue, Jinyu Li, Yifan Gong","year":"2013","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zaremba et al., 2014] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regulariza- tion. arXiv preprint arXiv:1409.2329, 2014.","title":"Recurrent neural network regularization","authors":"Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2016] Kai Zhang, Shandian Zhe, et al. An- nealed sparsity via adaptive and dynamic shrinking. In KDD, pages 1325-1334, 2016.","title":"Annealed sparsity via adaptive and dynamic shrinking","authors":"Kai Zhang, Shandian Zhe","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}