{"type":"ScholarlyArticle","title":"Regularizing Deep Neural Networks with an Ensemble-based Decorrelation Method","authors":[{"name":"Shuqin Gu","slug":"shuqin-gu","given":"Shuqin","family":"Gu","affiliation":"School of Computer Science and Technology , Tianjin University , Tianjin , China"},{"name":"Yuexian Hou","slug":"yuexian-hou","given":"Yuexian","family":"Hou","affiliation":"School of Computer Science and Technology , Tianjin University , Tianjin , China"},{"name":"Lipeng Zhang","slug":"lipeng-zhang","given":"Lipeng","family":"Zhang","affiliation":"School of Computer Software , Tianjin University , Tianjin , China"},{"name":"Yazhou Zhang","slug":"yazhou-zhang","given":"Yazhou","family":"Zhang","affiliation":"School of Computer Science and Technology , Tianjin University , Tianjin , China"}],"abstract":"Although Deep Neural Networks (DNNs) have achieved excellent performance in many tasks, improving the generalization capacity of DNNs still remains a challenge. In this work, we propose a novel regularizer named Ensemble-based Decorrelation Method (EDM), which is motivated by the idea of the ensemble learning to improve generalization capacity of DNNs. EDM can be applied to hidden layers in fully connected neural networks or convolutional neural networks. We treat each hidden layer as an ensemble of several base learners through dividing all the hidden units into several non-overlap groups, and each group will be viewed as a base learner. EDM encourages DNNs to learn more diverse representations by minimizing the covariance between all base learners during the training step. Experimental results on MNIST and CIFAR datasets demonstrate that EDM can effectively reduce the overfitting and improve the generalization capacity of DNNs","doi":"10.24963/ijcai.2018/301","url":"https://ijcai.org/proceedings/2018/301","pdf":"https://ijcai.org/proceedings/2018/0301.pdf","markdown":"https://ijcai.org/proceedings/2018/301.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/301","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":301,"pages":{"first":2177,"last":2183},"keywords":["Machine Learning: Neural Networks","Machine Learning: Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arX- iv:1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio and Bergstra, 2009] Yoshua Bengio and James S Bergstra. Slow, decorrelated features for pretraining com- plex cell-like networks. In Advances in neural information processing systems, pages 99-107, 2009.","title":"Slow, decorrelated features for pretraining complex cell-like networks","authors":"Yoshua Bengio, James S Bergstra","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chandar et al., 2016] Sarath Chandar, Mitesh M Khapra, Hugo Larochelle, and Balaraman Ravindran. Correlation- al neural networks. Neural computation, 2016.","title":"Correlational neural networks","authors":"Sarath Chandar, M Mitesh, Hugo Khapra, Balaraman Larochelle, Ravindran","year":"2016","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Yunpeng Chen, Xiaojie Jin, Jiashi Feng, and Shuicheng Yan. Training group orthogonal neural net- works with privileged information. arXiv preprint arX- iv:1701.06772, 2017.","title":"Training group orthogonal neural networks with privileged information","authors":"Yunpeng Chen, Xiaojie Jin, Jiashi Feng, Shuicheng Yan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/212"},{"position":5,"raw":"[Cheung et al., 2014] Brian Cheung, Jesse A Livezey, Ar- jun K Bansal, and Bruno A Olshausen. Discovering hid- den factors of variation in deep networks. arXiv preprint arXiv:1412.6583, 2014.","title":"Discovering hidden factors of variation in deep networks","authors":"Brian Cheung, Jesse A Livezey, Arjun K Bansal, Bruno A Olshausen","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cogswell et al., 2015] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representa- tions. arXiv preprint arXiv:1511.06068, 2015.","title":"Reducing overfitting in deep networks by decorrelating representations","authors":"Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, Dhruv Batra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geman et al., 1992] Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.","title":"Neural networks and the bias/variance dilemma","authors":"Stuart Geman, Elie Bienenstock, René Doursat","year":"1992","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, vol- ume 1. MIT press Cambridge, 2016.","title":"Deep learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In Acoustics, speech and signal pro- cessing (icassp), 2013 ieee international conference on, pages 6645-6649. IEEE, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"Acoustics, speech and signal processing (icassp), 2013 ieee international conference on","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jozefowicz et al., 2016] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Explor- ing the limits of language modeling. arXiv preprint arX- iv:1602.02410, 2016.","title":"Exploring the limits of language modeling","authors":"Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krogh and Hertz, 1992] Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Ad- vances in neural information processing systems, pages 950-957, 1992.","title":"A simple weight decay can improve generalization","authors":"Anders Krogh, John A Hertz","year":"1992","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levine et al., 2017] Yoav Levine, David Yakira, Nadav Co- hen, and Amnon Shashua. Deep learning and quantum en- tanglement: Fundamental connections with implications to network design. CoRR, abs/1704.01552, 2017.","title":"Deep learning and quantum entanglement: Fundamental connections with implications to network design","authors":"Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rodríguez et al., 2016] Pau Rodríguez, Jordi Gonzàlez, Guillem Cucurull, Josep M Gonfaus, and Xavier Roca. Regularizing cnns with locally constrained decorrelations. arXiv preprint arXiv:1611.01967, 2016.","title":null,"authors":"Rodríguez","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ueda and Nakano, 1996] Naonori Ueda and Ryohei Nakano. Generalization error of ensemble estimators. In Neural Networks, 1996., IEEE International Conference on, volume 1, pages 90-95. IEEE, 1996.","title":"Generalization error of ensemble estimators","authors":"Nakano Ueda, Naonori Ueda, Ryohei Nakano","year":"1996","venue":"IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wan et al., 2013] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International Conference on Machine Learning, pages 1058-1066, 2013.","title":"Regularization of neural networks using dropconnect","authors":"Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, Rob Fergus","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv:1609.08144, 2016.","title":"Google's neural machine translation system: Bridging the gap between human and machine translation","authors":"Yonghui Wu, Mike Schuster, Zhifeng Chen, V Quoc, Mohammad Le, Wolfgang Norouzi, Maxim Macherey, Yuan Krikun, Qin Cao, Klaus Gao, Macherey","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2017] Bingzhe Wu, Zhichao Liu, Zhihang Yuan, Guangyu Sun, and Charles Wu. Reducing overfitting in deep convolutional neural networks using redundancy reg- ularizer. In International Conference on Artificial Neural Networks, pages 49-55. Springer, 2017.","title":"Reducing overfitting in deep convolutional neural networks using redundancy regularizer","authors":"Wu","year":"2017","venue":"International Conference on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xie et al., 2015] Mengjiao Xie, Yuexian Hou, Peng Zhang, Jingfei Li, Wenjie Li, and Dawei Song. Modeling quantum entanglements in quantum language models. 2015.","title":"Modeling quantum entanglements in quantum language models","authors":"Mengjiao Xie, Yuexian Hou, Peng Zhang, Jingfei Li, Wenjie Li, Dawei Song","year":"2015","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2015/196"},{"position":22,"raw":"[Xiong et al., 2016] Wei Xiong, Bo Du, Lefei Zhang, Ruimin Hu, and Dacheng Tao. Regularizing deep con- volutional neural networks with a structured decorrelation constraint. In Data Mining (ICDM), 2016 IEEE 16th In- ternational Conference on, pages 519-528. IEEE, 2016.","title":"Regularizing deep convolutional neural networks with a structured decorrelation constraint","authors":"Wei Xiong, Bo Du, Lefei Zhang, Ruimin Hu, Dacheng Tao","year":"2016","venue":"IEEE 16th International Conference on","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zeiler and Fergus, 2014] Matthew D. Zeiler and Rob Fer- gus. Visualizing and understanding convolutional net- works. In In Computer Vision-ECCV 2014, pages 818- 833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"Matthew D Zeiler, Rob Fergus","year":"2014","venue":"Computer Vision-ECCV 2014","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou, 2012] Zhi-Hua Zhou. Ensemble methods: founda- tions and algorithms. CRC press, 2012.","title":"Ensemble methods: foundations and algorithms","authors":"Zhi-Hua Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}