{"type":"ScholarlyArticle","title":"EigenNet: Towards Fast and Structural Learning of Deep Neural Networks","authors":[{"name":"Ping Luo","slug":"ping-luo","given":"Ping","family":"Luo","affiliation":"Shenzhen Institutes of Advanced Technology , Chinese Academy of Sciences , Shenzhen , China; Department of Information Engineering , The Chinese University of Hong Kong , Hong Kong"}],"abstract":"Deep Neural Network (DNN) is difficult to train and easy to overfit in training. We address these two issues by introducing EigenNet, an architecture that not only accelerates training but also adjusts number of hidden neurons to reduce over-fitting. They are achieved by whitening the information flows of DNNs and removing those eigenvectors that may capture noises. The former improves conditioning of the Fisher information matrix, whilst the latter increases generalization capability. These appealing properties of EigenNet can benefit many recent DNN structures, such as network in network and inception, by wrapping their hidden layers into the layers of EigenNet. The modeling capacities of the original networks are preserved. Both the training wall-clock time and number of updates are reduced by using EigenNet, compared to stochastic gradient descent on various datasets, including MNIST, CIFAR-10, and CIFAR-100.","doi":"10.24963/ijcai.2017/338","url":"https://ijcai.org/proceedings/2017/338","pdf":"https://ijcai.org/proceedings/2017/0338.pdf","markdown":"https://ijcai.org/proceedings/2017/338.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/338","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":338,"pages":{"first":2428,"last":2434},"keywords":["Machine Learning: Deep Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Amari and Nagaoka, 2000] Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry. In Tanslations of Mathematical Monographs, 2000.","title":"Methods of information geometry","authors":"Nagaoka ; Ichi Amari, Hiroshi Amari, Nagaoka","year":"2000","venue":"Tanslations of Mathematical Monographs","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Desjardins et al., 2015] Guillaume Desjardins, Karen Si- monyan, Razvan Pascanu, and Koray Kavukcuoglu. Natural neural networks. In NIPS, 2015.","title":"Natural neural networks","authors":"Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, Koray Kavukcuoglu","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical Report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Larochelle et al., 2007] Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In ICML, 2007.","title":"An empirical evaluation of deep architectures on problems with many factors of variation","authors":"Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[LeCun et al., 1991] Yann LeCun, Ido Kanter, and Sara A. Sona. Second order properties of error surfaces: Learning time and generalization. In NIPS, 1991.","title":"Second order properties of error surfaces: Learning time and generalization","authors":"Yann Lecun, Ido Kanter, Sara A Sona","year":"1991","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[LeCun et al., 1993] Yann LeCun, Patrice Y. Simard, and Barak Pearlmutter. Automatic learning rate maximization by on-line estimation of the hessian's eigenvectors. In NIPS, 1993.","title":"Automatic learning rate maximization by on-line estimation of the hessian's eigenvectors","authors":"Yann Lecun, Patrice Y Simard, Barak Pearlmutter","year":"1993","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin et al., 2014] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.","title":"Network in network","authors":"Lin","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Martens and Grosse, 2015] James Martens and Roger Grosse. Optimizing neural networks with kronecker- factored approximate curvature. In ICML, 2015.","title":"Optimizing neural networks with kroneckerfactored approximate curvature","authors":"Grosse Martens, James Martens, Roger Grosse","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Raiko et al., 2012] Tapani Raiko, Harri Valpola, and Yann LeCun. Deep learning made easier by linear transforma- tions in perceptrons. In AISTATS, 2012.","title":"Deep learning made easier by linear transformations in perceptrons","authors":"Tapani Raiko, Harri Valpola, Yann Lecun","year":"2012","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Nicol N. Schraudolph. Centering neural network gradient factors. In Neural Networks: Tricks of the Trade, 2002.","title":"Centering neural network gradient factors","authors":"N Nicol, Schraudolph","year":"2002","venue":"Neural Networks: Tricks of the Trade","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Schraudolph, 2002b] Nicol N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. In Neural Computation, 2002.","title":"Fast curvature matrix-vector products for second-order gradient descent","authors":"N Nicol, Schraudolph","year":"2002","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. In Journal of Machine Learning Research, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Rmsprop: Divide the gradient by a running average of its recent magnitude. In Neural Networks for Machine Learning (Lecture 6.5), 2012.","title":"Rmsprop: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"Neural Networks for Machine Learning (Lecture 6","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}