{"type":"ScholarlyArticle","title":"Theoretical Investigation of Generalization Bound for Residual Networks","authors":[{"name":"Hao Chen","slug":"hao-chen","given":"Hao","family":"Chen","affiliation":"University of Science and Technology of China"},{"name":"Zhanfeng Mo","slug":"zhanfeng-mo","given":"Zhanfeng","family":"Mo","affiliation":"University of Science and Technology of China"},{"name":"Zhouwang Yang","slug":"zhouwang-yang","given":"Zhouwang","family":"Yang","affiliation":"University of Science and Technology of China"},{"name":"Xiao Wang","slug":"xiao-wang","given":"Xiao","family":"Wang","affiliation":"Purdue University"}],"abstract":"This paper presents a framework for norm-based capacity control with respect to an lp,q-norm in weight-normalized Residual Neural Networks (ResNets). We first formulate the representation of each residual block. For the regression problem, we analyze the Rademacher Complexity of the ResNets family. We also establish a tighter generalization upper bound for weight-normalized ResNets. in a more general sight. Using the lp,q-norm weight normalization in which 1/p+1/q >=1, we discuss the properties of a width-independent capacity control, which only relies on the depth according to a square root term. Several comparisons suggest that our result is tighter than previous work. Parallel results for Deep Neural Networks (DNN) and Convolutional Neural Networks (CNN) are included by introducing the lp,q-norm weight normalization for DNN and the lp,q-norm kernel normalization for CNN. Numerical experiments also verify that ResNet structures contribute to better generalization properties.","doi":"10.24963/ijcai.2019/288","url":"https://ijcai.org/proceedings/2019/288","pdf":"https://ijcai.org/proceedings/2019/0288.pdf","markdown":"https://ijcai.org/proceedings/2019/288.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/288","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":288,"pages":{"first":2081,"last":2087},"keywords":["Machine Learning: Probabilistic Machine Learning","Machine Learning: Deep Learning","Machine Learning Applications: Networks"],"videos":[],"references":[{"position":1,"raw":"[Bartlett et al., 2017] Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. In NIPS, 2017.","title":"Spectrally-normalized margin bounds for neural networks","authors":"L Peter, Dylan J Bartlett, Matus Foster, Telgarsky","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fan et al., 2018] Yuchen Fan, Jincheng Yu, and Thomas S. Huang. Wide-activated deep residual networks based restoration for bpg-compressed images. In CVPR Work- shops, 2018.","title":"Wide-activated deep residual networks based restoration for bpg-compressed images","authors":"Yuchen Fan, Jincheng Yu, Thomas S Huang","year":"2018","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Golowich et al., 2018a] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Pro- ceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 297-299. PMLR, 06-09 Jul 2018.","title":"Size-independent sample complexity of neural networks","authors":"Golowich","year":"2018","venue":"Proceedings of the 31st Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Golowich et al., 2018b] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In COLT, 2018.","title":"Size-independent sample complexity of neural networks","authors":"Golowich","year":"2018","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. 2016.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely con- nected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-2269, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2018] Xingguo Li, Junwei Lu, Zhaoran Wang, Jarvis Haupt, and Tuo Zhao. On tighter generalization bound for deep neural networks: Cnns, resnets, and be- yond. arXiv preprint arXiv:1806.05159, 2018.","title":"On tighter generalization bound for deep neural networks: Cnns, resnets, and beyond","authors":"Xingguo Li, Junwei Lu, Zhaoran Wang, Jarvis Haupt, Tuo Zhao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mo and Chen, 2019] Mo and Chen. IJCAI supplemen- tary material for theoretical investigation of generalization bound for residual networks. http://s000.tinyupload.com/ ?file id=52981413374013645026, 2019.","title":"IJCAI supplementary material for theoretical investigation of generalization bound for residual networks","authors":"Chen ; Mo, Chen Mo","year":"2019","venue":null,"doi":null,"url":"http://s000.tinyupload.com/?fileid=52981413374013645026,2019","internal_url":null},{"position":10,"raw":"[Mohri et al., 2012] Mehryar Mohri, Ameet Talwalkar, and Afshin Rostamizadeh. Foundations of machine learning (adaptive computation and machine learning series). Mit Press Cambridge, MA, 2012.","title":"Foundations of machine learning (adaptive computation and machine learning series)","authors":"Mehryar Mohri, Ameet Talwalkar, Afshin Rostamizadeh","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Neyshabur et al., 2015] Behnam Neyshabur, Ryota Tomio- ka, and Nathan Srebro. Norm-based capacity control in neural networks. In COLT, 2015.","title":"Norm-based capacity control in neural networks","authors":"Ryota Behnam Neyshabur, Nathan Tomioka, Srebro","year":"2015","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Neyshabur et al., 2017] Behnam Neyshabur, Srinadh Bho- janapalli, David McAllester, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. CoRR, abs/1707.09564, 2017.","title":"A pac-bayesian approach to spectrally-normalized margin bounds for neural networks","authors":"Srinadh Behnam Neyshabur, David Bhojanapalli, Nathan Mcallester, Srebro","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shalev-Shwartz and Ben-David, 2014] Shai Shalev- Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, New York, NY, USA, 2014.","title":"Understanding Machine Learning: From Theory to Algorithms","authors":"Shai Shalev, -Shwartz, Shai Ben-David","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Srivastava et al., 2015] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoR- R, abs/1505.00387, 2015.","title":"Highway networks","authors":"Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sun et al., 2016] Shizhao Sun, Wei Chen, Liwei Wang, Xi- aoguang Liu, and Tie-Yan Liu. On the depth of deep neural networks: A theoretical view. In AAAI, 2016.","title":"On the depth of deep neural networks: A theoretical view","authors":"Sun","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Xu and Wang, 2018] Yixi Xu and Xiao Wang. Understand- ing weight normalized deep neural networks with rectified linear units. In Advances in Neural Information Process- ing Systems, pages 130-139, 2018.","title":"Understanding weight normalized deep neural networks with rectified linear units","authors":"Wang ; ] Yixi Xu, Xiao Xu, Wang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}