{"type":"ScholarlyArticle","title":"AAR-CNNs: Auto Adaptive Regularized Convolutional Neural Networks","authors":[{"name":"Yao Lu","slug":"yao-lu","given":"Yao","family":"Lu","affiliation":"Shenzhen Graduate School , Harbin Institute of Technology , China"},{"name":"Guangming Lu","slug":"guangming-lu","given":"Guangming","family":"Lu","affiliation":"Shenzhen Graduate School , Harbin Institute of Technology , China"},{"name":"Yuanrong Xu","slug":"yuanrong-xu","given":"Yuanrong","family":"Xu","affiliation":"Shenzhen Graduate School , Harbin Institute of Technology , China"},{"name":"Bob Zhang","slug":"bob-zhang","given":"Bob","family":"Zhang","affiliation":"Department of Computer and Information Science , University of Macau , Macau yaolu"}],"abstract":"In order to address the overfitting problem caused by the small or simple training datasets and the large model’s size in Convolutional Neural Networks (CNNs), a novel Auto Adaptive Regularization (AAR) method is proposed in this paper. The relevant networks can be called AAR-CNNs. AAR is the first method using the “abstraction extent” (predicted by AE net) and a tiny learnable module (SE net) to auto adaptively predict more accurate and individualized regularization information. The AAR module can be directly inserted into every stage of any popular networks and trained end to end to improve the networks’ flexibility. This method can not only regularize the network at both the forward and the backward processes in the training phase, but also regularize the network on a more refined level (channel or pixel level) depending on the abstraction extent’s form. Comparative experiments are performed on low resolution ImageNet, CIFAR and SVHN datasets. Experimental results show that the AAR-CNNs can achieve state-of-the-art performances on these datasets.","doi":"10.24963/ijcai.2018/348","url":"https://ijcai.org/proceedings/2018/348","pdf":"https://ijcai.org/proceedings/2018/0348.pdf","markdown":"https://ijcai.org/proceedings/2018/348.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/348","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":348,"pages":{"first":2511,"last":2517},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Chrabaszcz et al., 2017] Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled vari- ant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017.","title":"A downsampled variant of imagenet as an alternative to the cifar datasets","authors":"Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pat- tern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, June 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630-645. Springer International Publishing, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hu et al., 2017] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 2017.","title":"Squeeze-and-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arX- iv:1608.06993v4, 2017.","title":"Densely connected convolutional networks","authors":"Huang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Larsson et al., 2016] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neu- ral networks without residuals. arXiv preprint arX- iv:1605.07648, 2016.","title":"Fractalnet: Ultra-deep neural networks without residuals","authors":"Larsson","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2015] Chen-Yu Lee, Saining Xie, Patrick Gal- lagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Artificial Intelligence and Statistics, pages 562-570, 2015.","title":"Deeplysupervised nets","authors":"Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu","year":"2015","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 5, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS workshop on deep learning and unsupervised feature learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nowlan and Hinton, 1992] Steven J Nowlan and Geoffrey E Hinton. Simplifying neural networks by soft weight- sharing. Neural computation, 4(4):473-493, 1992.","title":"Simplifying neural networks by soft weightsharing","authors":"J Steven, Geoffrey E Nowlan, Hinton","year":"1992","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Saad, 1998] David Saad. Online algorithms and stochastic approximations. Online Learning, 5, 1998.","title":"Online algorithms and stochastic approximations","authors":"David Saad","year":"1998","venue":"Online Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sermanet et al., 2012] Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Pattern Recogni- tion (ICPR), 2012 21st International Conference on, pages 3288-3291. IEEE, 2012.","title":"Convolutional neural networks applied to house numbers digit classification","authors":"Sermanet","year":"2012","venue":"21st International Conference on","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutskever et al., 2013a] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Interna- tional conference on machine learning, pages 1139-1147, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutskever et al., 2013b] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Interna- tional conference on machine learning, pages 1139-1147, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}