{"type":"ScholarlyArticle","title":"Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?","authors":[{"name":"Lifu Wang","slug":"lifu-wang","given":"Lifu","family":"Wang","affiliation":"Department of Electronic and Information Engineering , Beijing Jiaotong University , Beijing , China Key Laboratory of Communication and Information Systems ,"},{"name":"Bo Shen","slug":"bo-shen","given":"Bo","family":"Shen","affiliation":"Department of Electronic and Information Engineering , Beijing Jiaotong University , Beijing , China Key Laboratory of Communication and Information Systems ,"},{"name":"Ning Zhao","slug":"ning-zhao","given":"Ning","family":"Zhao","affiliation":"Department of Electronic and Information Engineering , Beijing Jiaotong University , Beijing , China State Key Lab of Rail Traffic Control and Safety ,"},{"name":"Zhiyuan Zhang","slug":"zhiyuan-zhang","given":"Zhiyuan","family":"Zhang","affiliation":"Department of Electronic and Information Engineering , Beijing Jiaotong University , Beijing , China Key Laboratory of Communication and Information Systems ,; Beijing Municipal Commission of Education Beijing Jiaotong University , Beijing , China; Beijing Jiaotong University , Beijing , China"}],"abstract":"The residual network is now one of the most effective structures in deep learning, which utilizes the skip connections to “guarantee\" the performance will not get worse. However, the non-convexity of the neural network makes it unclear whether the skip connections do provably improve the learning ability since the nonlinearity may create many local minima. In some previous works  [Freeman and Bruna, 2016], it is shown that despite the non-convexity, the loss landscape of the two-layer ReLU network has good properties when the number m of hidden nodes is very large. In this paper, we follow this line to study the topology (sub-level sets) of the loss landscape of deep ReLU neural networks with a skip connection and theoretically prove that the skip connection network inherits the good properties of the two-layer network and skip connections can  help to control the connectedness of the sub-level sets, such that any local minima worse than the global minima of some two-layer ReLU network will be very “shallow\". The “depth\" of these local minima are at most O(m^(η-1)/n), where n is the input dimension, η<1. This provides a theoretical explanation for the effectiveness of the skip connection in deep learning.","doi":"10.24963/ijcai.2020/387","url":"https://ijcai.org/proceedings/2020/387","pdf":"https://ijcai.org/proceedings/2020/0387.pdf","markdown":"https://ijcai.org/proceedings/2020/387.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/387","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":387,"pages":{"first":2792,"last":2798},"keywords":["Machine Learning: Learning Theory","Machine Learning: Deep Learning","Machine Learning: Deep-learning Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24935/file_24935.m3u8","length_sec":279,"provider":"hls"}],"references":[{"position":1,"raw":"[Allenzhu and Li, 2019] Zeyuan Allenzhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernel- s? Advances in Neural Information Processing Systems, pages 9015-9025, 2019.","title":"Zeyuan Allenzhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels","authors":"Li Allenzhu","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barber and Ha, 2018] Rina Foygel Barber and Wooseok Ha. Gradient descent with non-convex constraints: local con- cavity determines convergence. Information and Infer- ence: A Journal of the IMA, 7(4):755-806, 2018.","title":"Gradient descent with non-convex constraints: local concavity determines convergence","authors":"Rina Foygel Barber, Wooseok Ha","year":"2018","venue":"Information and Inference: A Journal of the IMA","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barron and A.R., 1993] Barron and A.R. Universal approx- imation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3), 1993.","title":"Universal approximation bounds for superpositions of a sigmoidal function","authors":"A R Barron","year":"1993","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Draxler et al., 2018] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A Hamprecht. Essentially no barriers in neural network energy landscape. Interna- tional conference on machine learning, pages 1308-1317, 2018.","title":"Essentially no barriers in neural network energy landscape. International conference on machine learning","authors":"Draxler","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du and Lee, 2018] Simon S Du and Jason D Lee. On the power of over-parametrization in neural networks with quadratic activation. International conference on machine learning, pages 1328-1337, 2018.","title":"On the power of over-parametrization in neural networks with quadratic activation","authors":"S Simon, Jason D Du, Lee","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Du et al., 2018] Simon S Du, Jason D Lee, and Yuandong Tian. When is a convolutional filter easy to learn. Interna- tional conference on machine learning, 2018.","title":"When is a convolutional filter easy to learn","authors":"Jason D Simon S Du, Yuandong Lee, Tian","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Freeman and Bruna, 2016] C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. International conference on machine learn- ing, 2016.","title":"Topology and geometry of half-rectified network optimization","authors":"C Daniel, Freeman, Joan Bruna","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Garipov et al., 2018] Timur Garipov, Pavel Izmailov, D- mitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensem- bling of dnns. Advances in neural information processing systems, pages 8789-8798, 2018.","title":"Loss surfaces, mode connectivity, and fast ensembling of dnns","authors":"Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon, Wilson","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feed- forward neural networks. International conference on ar- tificial intelligence and statistics, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ha et al., 2018] Wooseok Ha, Haoyang Liu, and Ri- na Foygel Barber. An equivalence between stationary points for rank constraints versus low-rank factorization- s. arXiv: Optimization and Control, 2018.","title":"and Rina Foygel Barber. An equivalence between stationary points for rank constraints versus low-rank factorizations","authors":"Ha","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016a] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. European conference on computer vision, pages 630-645, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2018] Chi Jin, Lydia T Liu, Rong Ge, and Michael I Jordan. On the local minima of the empirical risk. Advances in neural information processing systems, pages 4896-4905, 2018.","title":"On the local minima of the empirical risk","authors":"Jin","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kawaguchi and Bengio, 2019] Kenji Kawaguchi and Yoshua Bengio. Depth with nonlinearity creates no bad local minima in resnets. Neural Networks, 118:167-174, 2019.","title":"Depth with nonlinearity creates no bad local minima in resnets","authors":"Bengio ; ] Kenji Kawaguchi, Yoshua Kawaguchi, Bengio","year":"2019","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kawaguchi, 2016] Kenji Kawaguchi. Deep learning with- out poor local minima. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 586- 594.","title":"Deep learning without poor local minima","authors":"Kenji Kawaguchi","year":null,"venue":"Advances in Neural Information Processing Systems 29","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolution- al neural networks. Advances in neural information pro- cessing systems, 25(2), 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, I Sutskever, G Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kuditipudi et al., 2019] Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in Neu- ral Information Processing Systems, pages 14574-14583, 2019.","title":"Explaining landscape connectivity of low-cost solutions for multilayer nets","authors":"Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, Sanjeev Arora","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2018] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Pro- cessing Systems 31, pages 6389-6399.","title":"Visualizing the loss landscape of neural nets","authors":"Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Curran Associates, Inc., 2018.","title":null,"authors":"Curran Associates, Inc","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mahdi et al., 2018] Soltanolkotabi Mahdi, Javanmard Adel, and Lee Jason D. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 2018.","title":"Theoretical insights into the optimization landscape of over-parameterized shallow neural networks","authors":"Mahdi","year":"2018","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Safran and Shamir, 2018] Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neu- ral networks. International conference on machine learn- ing, pages 4430-4438, 2018.","title":"Spurious local minima are common in two-layer relu neural networks","authors":"Itay Safran, Ohad Shamir","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shamir, 2018] Ohad Shamir. Are resnets provably better than linear predictors. Advances in neural information pro- cessing systems, pages 507-516, 2018.","title":"Are resnets provably better than linear predictors","authors":"Ohad Shamir","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tian, 2017] Yuandong Tian. Symmetry-breaking conver- gence analysis of certain two-layered neural networks with relu nonlinearity. International conference on learning representations, 2017.","title":"Symmetry-breaking convergence analysis of certain two-layered neural networks with relu nonlinearity","authors":"Yuandong Tian, Tian","year":"2017","venue":"International conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Veit et al., 2016] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of rel- atively shallow networks. Advances in Neural Information Processing Systems, pages 550-558, 2016.","title":"Residual networks behave like ensembles of relatively shallow networks","authors":"Andreas Veit, Michael J Wilber, Serge Belongie","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yun et al., 2019] Chulhee Yun, Suvrit Sra, and Ali Jad- babaie. Are deep resnets provably better than linear predic- tors. Advances in Neural Information Processing Systems, 2019.","title":"Ali Jadbabaie. Are deep resnets provably better than linear predictors","authors":"Yun","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Yuchen Zhang, Percy Liang, and Moses Charikar. A hitting time analysis of stochastic gradien- t langevin dynamics. Conference on Learning Theory, 2017.","title":"A hitting time analysis of stochastic gradient langevin dynamics","authors":"Yuchen Zhang, Percy Liang, Moses Charikar","year":"2017","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}