{"type":"ScholarlyArticle","title":"Doubly Sparsifying Network","authors":[{"name":"Zhangyang Wang","slug":"zhangyang-wang","given":"Zhangyang","family":"Wang","affiliation":"Department of Computer Science and Engineering , Michigan State University Beckman Institute , University of Illinois at Urbana-Champaign"},{"name":"Shuai Huang","slug":"shuai-huang","given":"Shuai","family":"Huang","affiliation":"Department of Computer Science and Engineering , Michigan State University Beckman Institute , University of Illinois at Urbana-Champaign"},{"name":"Jiayu Zhou","slug":"jiayu-zhou","given":"Jiayu","family":"Zhou","affiliation":"§ †Department of Computer Science and Engineering , Texas A&M University ‡Department of Industrial and Systems Engineering , University of Washington"},{"name":"Thomas S. Huang","slug":"thomas-s-huang","given":"Thomas S.","family":"Huang","affiliation":"Department of Computer Science and Engineering , Michigan State University Beckman Institute , University of Illinois at Urbana-Champaign"}],"abstract":"We propose the doubly sparsifying network (DSN), by drawing inspirations from the double sparsity model for dictionary learning. DSN emphasizes the joint utilization of both the problem structure and the parameter structure. It simultaneously sparsifies the output features and the learned model parameters, under one unified framework. DSN enjoys intuitive model interpretation, compact model size and low complexity. We compare DSN against a few carefully-designed baselines, to verify its consistently superior performance in a wide range of settings. Encouraged by its robustness to insufficient training data, we explore the applicability of DSN in brain signal processing that has been a challenging interdisciplinary area. DSN is evaluated for two mainstream tasks, electroencephalographic (EEG) signal classification and blood oxygenation level dependent (BOLD) response prediction, both achieving promising results.","doi":"10.24963/ijcai.2017/421","url":"https://ijcai.org/proceedings/2017/421","pdf":"https://ijcai.org/proceedings/2017/0421.pdf","markdown":"https://ijcai.org/proceedings/2017/421.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/421","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":421,"pages":{"first":3020,"last":3026},"keywords":["Machine Learning: Data Mining","Machine Learning: Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 1998] Charles W Anderson, Erik A Stolz, and Sanyogita Shamsunder. Multivariate autoregressive models for classification of spontaneous electroencephalo- graphic signals during mental tasks. IEEE Transactions on Biomedical Engineering, 45(3):277-286, 1998.","title":"Multivariate autoregressive models for classification of spontaneous electroencephalographic signals during mental tasks","authors":"Erik A Charles W Anderson, Sanyogita Stolz, Shamsunder","year":"1998","venue":"IEEE Transactions on Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blumensath and Davies, 2008] Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approx- imations. Journal of Fourier Analysis and Applications, 2008.","title":"Iterative thresholding for sparse approximations","authors":"Thomas Blumensath, Mike E Davies","year":"2008","venue":"Journal of Fourier Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bottou, 2010] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP- STAT'2010. Springer, 2010.","title":"Large-scale machine learning with stochastic gradient descent","authors":"Léon Bottou, Bottou","year":"2010","venue":"Proceedings of COMP-STAT'2010","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brodu et al., 2011] Nicolas Brodu, Fabien Lotte, and Ana- tole Lécuyer. Comparative study of band-power extraction techniques for motor imagery classification. In Computa- tional Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), 2011 IEEE Symposium on, 2011.","title":"Comparative study of band-power extraction techniques for motor imagery classification","authors":"Brodu","year":"2011","venue":"Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ciresan et al., 2010] DC Ciresan, U Meier, LM Gam- bardella, and J Schmidhuber. Deep big simple neural nets excel on handwritten digit recognition. 2010.","title":"Deep big simple neural nets excel on handwritten digit recognition","authors":"Ciresan","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Coates and Ng, 2011] Adam Coates and Andrew Y Ng. The importance of encoding versus training with sparse coding and vector quantization. In ICML, 2011.","title":"The importance of encoding versus training with sparse coding and vector quantization","authors":"Ng ; ] Adam Coates, Andrew Y Coates, Ng","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Elad and Aharon, 2006] Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE TIP, 2006.","title":"Image denoising via sparse and redundant representations over learned dictionaries","authors":"Aharon ; Elad, Michael Elad, Michal Aharon","year":"2006","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Erhan et al., 2010] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? JMLR, 2010.","title":"Why does unsupervised pre-training help deep learning? JMLR","authors":"Yoshua Dumitru Erhan, Aaron Bengio, Pierre-Antoine Courville, Pascal Manzagol, Samy Vincent, Bengio","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gregor and LeCun, 2010] Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In ICML, 2010.","title":"Learning fast approximations of sparse coding","authors":"Karol Gregor, Yann Lecun","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hung et al., 2005] Chih-I Hung, Po-Lei Lee, Yu-Te Wu, Li- Fen Chen, Tzu-Chen Yeh, and Jen-Chuen Hsieh. Recogni- tion of motor imagery electroencephalography using inde- pendent component analysis and machine classifiers. An- nals of Biomedical Engineering, 33(8):1053-1070, 2005.","title":"Recognition of motor imagery electroencephalography using independent component analysis and machine classifiers","authors":"Chih-I Hung, Po-Lei Lee, Yu-Te Wu, Li-Fen Chen, Tzu-Chen Yeh, Jen-Chuen Hsieh","year":"2005","venue":"Annals of Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2016] Xiaojie Jin, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint arXiv:1607.05423, 2016.","title":"Training skinny deep neural networks with iterative hard thresholding methods","authors":"Xiaojie Jin, Xiaotong Yuan, Jiashi Feng, Shuicheng Yan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kay et al., 2013] Kendrick N Kay, Jonathan Winawer, Ariel Rokem, Aviv Mezer, and Brian A Wandell. A two-stage cascade model of bold responses in human visual cortex. PLoS Comput Biol, 9(5):e1003079, 2013.","title":"A two-stage cascade model of bold responses in human visual cortex","authors":"Jonathan Kendrick N Kay, Ariel Winawer, Aviv Rokem, Brian A Mezer, Wandell","year":"2013","venue":"PLoS Comput Biol","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Meier et al., 2011] Ueli Meier, Dan Claudiu Ciresan, Luca Maria Gambardella, and Jurgen Schmidhuber. Better digit recognition with a committee of simple neural nets. In ICDAR. IEEE, 2011.","title":"Better digit recognition with a committee of simple neural nets","authors":"Ueli Meier, Dan Claudiu Ciresan, Luca, Maria Gambardella, Jurgen Schmidhuber","year":"2011","venue":"ICDAR. IEEE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Neyshabur and Panigrahy, 2013] Behnam Neyshabur and Rina Panigrahy. Sparse matrix factorization. arXiv preprint arXiv:1311.3315, 2013.","title":"Sparse matrix factorization","authors":"Panigrahy Neyshabur, Behnam Neyshabur, Rina Panigrahy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Olshausen and Field, 1997] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311- 3325, 1997.","title":"Sparse coding with an overcomplete basis set: A strategy employed by v1?","authors":"A Bruno, David J Olshausen, Field","year":"1997","venue":"Vision research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren and Wu, 2014] Yuanfang Ren and Yan Wu. Convolu- tional deep belief networks for feature extraction of eeg signal. In IJCNN. IEEE, 2014.","title":"Convolutional deep belief networks for feature extraction of eeg signal","authors":"Ren, Wu ; ] Yuanfang Ren, Yan Wu","year":"2014","venue":"IJCNN. IEEE","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rubinstein et al., 2010] Ron Rubinstein, Michael Zibulevsky, and Michael Elad. Double sparsity: Learning sparse dictionaries for sparse signal approximation. IEEE Transactions on signal processing, 58(3):1553-1564, 2010.","title":"Double sparsity: Learning sparse dictionaries for sparse signal approximation","authors":"Ron Rubinstein, Michael Zibulevsky, Michael Elad","year":"2010","venue":"IEEE Transactions on signal processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schlögl, 2003] Alois Schlögl. Outcome of the bci- competition 2003 on the graz data set. Berlin, Germany: Graz University of Technology, 2003.","title":"Outcome of the bcicompetition 2003 on the graz data set","authors":"Alois Schlögl, Schlögl","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sprechmann et al., 2015] Pablo Sprechmann, Alexander Bronstein, and Guillermo Sapiro. Learning efficient sparse and low rank models. IEEE TPAMI, 2015.","title":"Learning efficient sparse and low rank models","authors":"Sprechmann","year":"2015","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tabar and Halici, 2016] Yousef Rezaei Tabar and Ugur Halici. A novel deep learning approach for classification of eeg motor imagery signals. Journal of Neural Engineering, 14(1):016003, 2016.","title":"A novel deep learning approach for classification of eeg motor imagery signals","authors":"Yousef Rezaei, Tabar, Ugur Halici","year":"2016","venue":"Journal of Neural Engineering","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wan et al., 2013] Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 1058-1066, 2013.","title":"Regularization of neural networks using dropconnect","authors":"Li Wan, Matthew Zeiler, Sixin Zhang, Rob Yann L Cun, Fergus","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning (ICML-13)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016a] Zhangyang Wang, Shiyu Chang, Jiayu Zhou, Meng Wang, and Thomas S Huang. Learning a task- specific deep architecture for clustering. In SDM, 2016.","title":"Learning a taskspecific deep architecture for clustering","authors":"Wang","year":"2016","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2016b] Zhangyang Wang, Qing Ling, and Thomas Huang. Learning deep ℓ 0 encoders. AAAI, 2016.","title":"Learning deep ℓ 0 encoders","authors":"Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2016c] Zhangyang Wang, Ding Liu, Shiyu Chang, Qing Ling, Yingzhen Yang, and Thomas S Huang. D3: Deep dual-domain based fast restoration of jpeg- compressed images. In IEEE CVPR, 2016.","title":"D3: Deep dual-domain based fast restoration of jpegcompressed images","authors":"Wang","year":"2016","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2016d] Zhangyang Wang, Yingzhen Yang, Shiyu Chang, Qing Ling, and Thomas S Huang. Learning a deep l? encoder for hashing. In IJCAI, 2016.","title":"Learning a deep l? encoder for hashing","authors":"Wang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/310"},{"position":29,"raw":"[Wolpaw et al., 2000] Jonathan R Wolpaw, Niels Birbaumer, William J Heetderks, Dennis J McFarland, P Hunter Peck- ham, Gerwin Schalk, Emanuel Donchin, Louis A Quatrano, Charles J Robinson, Theresa M Vaughan, et al. Brain- computer interface technology: a review of the first in- ternational meeting. IEEE transactions on rehabilitation engineering, 8(2):164-173, 2000.","title":"Braincomputer interface technology: a review of the first international meeting","authors":"Niels Jonathan R Wolpaw, William J Birbaumer, Dennis J Heetderks, P Mcfarland, Gerwin Hunter Peckham, Emanuel Schalk, Louis A Donchin, Charles J Quatrano, Theresa M Robinson, Vaughan","year":"2000","venue":"IEEE transactions on rehabilitation engineering","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xin et al., 2016] Bo Xin, Yizhou Wang, Wen Gao, and David Wipf. Maximal sparsity with deep networks? NIPS, 2016.","title":"Maximal sparsity with deep networks? NIPS","authors":"Bo Xin, Yizhou Wang, Wen Gao, David Wipf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}