{"type":"ScholarlyArticle","title":"Building Sparse Deep Feedforward Networks using Tree Receptive Fields","authors":[{"name":"Xiaopeng Li","slug":"xiaopeng-li","given":"Xiaopeng","family":"Li","affiliation":"Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay , Hong Kong"},{"name":"Zhourong Chen","slug":"zhourong-chen","given":"Zhourong","family":"Chen","affiliation":"Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay , Hong Kong"},{"name":"Nevin L. Zhang","slug":"nevin-l-zhang","given":"Nevin L.","family":"Zhang","affiliation":"Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay , Hong Kong"}],"abstract":"Sparse connectivity is an important factor behind the success of convolutional neural networks and recurrent neural networks. In this paper, we consider the problem of learning sparse connectivity for feedforward neural networks (FNNs). The key idea is that a unit should be connected to a small number of units at the next level below that are strongly correlated. We use Chow-Liu's algorithm to learn a tree-structured probabilistic model for the units at the current level, use the tree to identify subsets of units that are strongly correlated, and introduce a new unit with receptive field over the subsets. The procedure is repeated on the new units to build multiple layers of hidden units. The resulting model is called a TRF-net. Empirical results show that, when compared to dense FNNs, TRF-net achieves better or comparable classification performance with much fewer parameters and sparser structures. They are also more interpretable.","doi":"10.24963/ijcai.2018/700","url":"https://ijcai.org/proceedings/2018/700","pdf":"https://ijcai.org/proceedings/2018/0700.pdf","markdown":"https://ijcai.org/proceedings/2018/700.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/700","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Uncertainty in AI","track":"Main track","number":700,"pages":{"first":5045,"last":5051},"keywords":["Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Neural Networks","Uncertainty in AI: Graphical Models"],"videos":[],"references":[{"position":1,"raw":"[Adams et al., 2010] Ryan Adams, Hanna Wallach, and Zoubin Ghahramani. Learning the structure of deep sparse graphical models. In Proceedings of the Thirteenth Inter- national Conference on Artificial Intelligence and Statis- tics, pages 1-8, 2010.","title":"Learning the structure of deep sparse graphical models","authors":"Ryan Adams, Hanna Wallach, Zoubin Ghahramani","year":"2010","venue":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ash, 1989] Timur Ash. Dynamic node creation in back- propagation networks. Connection science, 1(4):365-375, 1989.","title":"Dynamic node creation in backpropagation networks","authors":"Ash ; Timur Ash","year":"1989","venue":"Connection science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baker et al., 2017] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network ar- chitectures using reinforcement learning. In Proceedings of the 5th International Conference on Learning Represen- tations, 2017.","title":"Designing neural network architectures using reinforcement learning","authors":"Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar","year":"2017","venue":"Proceedings of the 5th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bello, 1992] Martin G Bello. Enhanced training algorithms, and integrated training/architecture selection for multi- layer perceptron networks. IEEE Transactions on Neural networks, 3(6):864-875, 1992.","title":"Enhanced training algorithms, and integrated training/architecture selection for multi-layer perceptron networks","authors":"G Martin, Bello","year":"1992","venue":"IEEE Transactions on Neural networks","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2017] Zhourong Chen, Nevin L Zhang, Dit-Yan Yeung, and Peixian Chen. Sparse boltzmann machines with structure learning as applied to text analysis. In Pro- ceedings of the 31st AAAI Conference on Artificial Intelli- gence, 2017.","title":"Sparse boltzmann machines with structure learning as applied to text analysis","authors":"Chen","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chow and Liu, 1968] C Chow and Cong Liu. Approx- imating discrete probability distributions with depen- dence trees. IEEE transactions on Information Theory, 14(3):462-467, 1968.","title":"Approximating discrete probability distributions with dependence trees","authors":"C Chow, Cong Liu","year":"1968","venue":"IEEE transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Collins and Kohli, 2014] Maxwell D Collins and Pushmeet Kohli. Memory bounded deep convolutional networks. arXiv preprint arXiv:1412.1442, 2014.","title":"Memory bounded deep convolutional networks","authors":"D Maxwell, Pushmeet Collins, Kohli","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fahlman and Lebiere, 1990] Scott E Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In Advances in Neural Information Processing Systems, pages 524-532, 1990.","title":"The cascade-correlation learning architecture","authors":"E Scott, Christian Fahlman, Lebiere","year":"1990","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2016] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Ad- vances in Neural Information Processing Systems, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Yiwen Guo, Anbang Yao, Yurong Chen","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Informa- tion Processing Systems, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hassibi et al., 1993] Babak Hassibi, David G. Stork, and Stork Crc. Ricoh. Com. Second order derivatives for net- work pruning: Optimal brain surgeon. In Advances in Neu- ral Information Processing Systems, 1993.","title":"Second order derivatives for network pruning: Optimal brain surgeon","authors":"Babak Hassibi, David G Stork, Stork Crc, Ricoh, Com","year":"1993","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hsu et al., 2017] Wilson Hsu, Agastya Kalra, and Pas- cal Poupart. Online structure learning for sum- product networks with gaussian leaves. arXiv preprint arXiv:1701.05265, 2017.","title":"Online structure learning for sumproduct networks with gaussian leaves","authors":"Wilson Hsu, Agastya Kalra, Pascal Poupart","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceed- ings of the 3rd International Conference on Learning Rep- resentations, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"Proceedings of the 3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klambauer et al., 2017] Günter Klambauer, Thomas Un- terthiner, Andreas Mayr, and Sepp Hochreiter. Self- normalizing neural networks. In Advances in Neural In- formation Processing Systems, 2017.","title":"Selfnormalizing neural networks","authors":"Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kwok and Yeung, 1997a] Tin-Yan Kwok and Dit-Yan Ye- ung. Objective functions for training new hidden units in constructive neural networks. IEEE Transactions on neu- ral networks, 8(5):1131-1148, 1997.","title":"Objective functions for training new hidden units in constructive neural networks","authors":"Yeung ; A] Tin-Yan Kwok, Dit-Yan Kwok, Yeung","year":"1997","venue":"IEEE Transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kwok and Yeung, 1997b] Tin-Yau Kwok and Dit-Yan Ye- ung. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Transactions on Neural Networks, 8(3):630-645, 1997.","title":"Tin-Yau Kwok and Dit-Yan Yeung. Constructive algorithms for structure learning in feedforward neural networks for regression problems","authors":"Yeung Kwok","year":"1997","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 1990] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1990.","title":"Optimal brain damage","authors":"Yann Lecun, John S Denker, Sara A Solla","year":"1990","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2017] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In Proceedings of the 5rd International Conference on Learning Representations, 2017.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2017","venue":"Proceedings of the 5rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2017] Jia Liu, Maoguo Gong, Qiguang Miao, Xi- aogang Wang, and Hao Li. Structure learning for deep neural networks based on multiobjective optimization. IEEE Transactions on Neural Networks and Learning Sys- tems, 2017.","title":"Structure learning for deep neural networks based on multiobjective optimization","authors":"Jia Liu, Maoguo Gong, Qiguang Miao, Xiaogang Wang, Hao Li","year":"2017","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In International Confer- ence on Learning Representations Workshops, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":"International Conference on Learning Representations Workshops","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their composi- tionality. In Advances in Neural Information Processing Systems, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann ma- chines. In Proceedings of the International Conference on Machine Learning, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Srinivas and Babu, 2015] Suraj Srinivas and R. Venkatesh Babu. Data-free parameter pruning for deep neural net- works. In Proceedings of the British Machine Vision Con- ference, 2015.","title":"Data-free parameter pruning for deep neural networks","authors":"Suraj Srinivas, R Venkatesh, Babu","year":"2015","venue":"Proceedings of the British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extract- ing and composing robust features with denoising autoen- coders. In Proceedings of the 25th International confer- ence on Machine learning, pages 1096-1103. ACM, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":"Proceedings of the 25th International conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, pages 2074-2082, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Process- ing Systems, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In Pro- ceedings of the 5th International Conference on Learning Representations, 2017.","title":"Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2017","venue":"Proceedings of the 5th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}