{"type":"ScholarlyArticle","title":"BN-invariant Sharpness Regularizes the Training Model to Better Generalization","authors":[{"name":"Mingyang Yi","slug":"mingyang-yi","given":"Mingyang","family":"Yi","affiliation":"University of Chinese Academy of Sciences; Academy of Mathematics and Systems Science"},{"name":"Huishuai Zhang","slug":"huishuai-zhang","given":"Huishuai","family":"Zhang","affiliation":"Microsoft Research"},{"name":"Wei Chen","slug":"wei-chen","given":"Wei","family":"Chen","affiliation":"Microsoft Research"},{"name":"Zhi-Ming Ma","slug":"zhi-ming-ma","given":"Zhi-Ming","family":"Ma","affiliation":"University of Chinese Academy of Sciences; Academy of Mathematics and Systems Science"},{"name":"Tie-Yan Liu","slug":"tie-yan-liu","given":"Tie-Yan","family":"Liu","affiliation":"Microsoft Research"}],"abstract":"It is arguably believed that flatter minima can generalize better. However, it has been pointed out that the usual definitions of sharpness, which consider either the maxima or the integral of loss over a delta ball of parameters around minima, cannot give consistent measurement for scale invariant neural networks, e.g., networks with batch normalization layer. In this paper, we first propose a measure of sharpness, BN-Sharpness, which gives consistent value for equivalent networks under BN. \r\nIt achieves the property of scale invariance by connecting the integral diameter with the scale of parameter. Then we present a computation-efficient way to calculate the BN-sharpness approximately i.e., one dimensional integral along the \"sharpest\" direction. Furthermore, we use the BN-sharpness to regularize the training and design an algorithm to minimize the new regularized objective. Our algorithm achieves considerably better performance than vanilla SGD over various  experiment settings.","doi":"10.24963/ijcai.2019/578","url":"https://ijcai.org/proceedings/2019/578","pdf":"https://ijcai.org/proceedings/2019/0578.pdf","markdown":"https://ijcai.org/proceedings/2019/578.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/578","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":578,"pages":{"first":4164,"last":4170},"keywords":["Machine Learning: Deep Learning","Machine Learning: Explainable Machine Learning","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Allen-Zhu et al., 2018] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. arXiv preprint arXiv:, 2018.","title":"A convergence theory for deep learning via over-parameterization","authors":"Allen-Zhu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boumal et al., 2017] Nicolas Boumal, Absil Pierre- Antoine, and Coralia Cartis. Global rates of convergence for nonconvex optimization on manifolds. arxiv preprint arxiv:1605.08101, 2017.","title":"Global rates of convergence for nonconvex optimization on manifolds","authors":"Boumal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhari et al., 2017] Pratik Chaudhari, Anna Choroman- ska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In ICLR'2017 arXiv:1611.01838, 2017.","title":"Entropy-sgd: Biasing gradient descent into wide valleys","authors":"Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann Lecun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, Riccardo Zecchina","year":"2017","venue":"ICLR'2017","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho and Lee, 2018] Minhyung Cho and Jaehyung Lee. Rie- mannian approach to batch normalization. arXiv preprint arXiv:1709.09603, 2018.","title":"Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization","authors":"Lee Cho","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choromanska et al., 2015] Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In- ternational Conference on Artificial Intelligence and Statistics(AISTATS 2015), 2015.","title":"The loss surfaces of multilayer networks","authors":"Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, Yann Lecun","year":"2015","venue":"ternational Conference on Artificial Intelligence and Statistics(AISTATS 2015)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[De et al., 2017] Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. International Conference on Artificial Intelli- gence and Statistics(AISTATS 2017), 2017.","title":"Automated inference with adaptive batches","authors":"Soham De, Abhay Yadav, David Jacobs, Tom Goldstein","year":"2017","venue":"International Conference on Artificial Intelligence and Statistics(AISTATS 2017)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dinh et al., 2017] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In ICML'2017 arXiv:1703.04933, 2017.","title":"Sharp minima can generalize for deep nets","authors":"Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio","year":"2017","venue":"ICML'2017","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Du et al., 2018] Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint arXiv:1811.03804, 2018.","title":"Gradient descent finds global minima of deep neural networks","authors":"Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, Xiyu Zhai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hoffer et al., 2017] Elad Hoffer, Itay Hubara, and Daniel Soudryn. Train longer, generalize better: closing the gen- eralization gap in large batch training of neural networks. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.","title":"Train longer, generalize better: closing the generalization gap in large batch training of neural networks","authors":"Elad Hoffer, Itay Hubara, Daniel Soudryn","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over mul- tiple dependent stiefel manifolds in deep neural networks. arXiv preprint arXiv:1709.06079, 2017.","title":"Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks","authors":"Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, Bo Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning,ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"Proceedings of The 32nd International Conference on Machine Learning,ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Keskar et al., 2017] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR'2017 arxiv: 1609.04836, 2017.","title":"On large-batch training for deep learning: Generalization gap and sharp minima","authors":"Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak, Peter Tang","year":"2017","venue":"ICLR'2017","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 1998] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278 -2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Lecun","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. arxiv preprint arXiv:1712.0991, 2018.","title":"Visualizing the loss landscape of neural nets","authors":"Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McAllesterl, 1998] David A. McAllesterl. Some pac- bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory, pages 230- 234, 1998.","title":"Some pacbayesian theorems","authors":"David A Mcallesterl","year":"1998","venue":"Proceedings of the eleventh annual conference on Computational learning theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McAllesterl, 1999] David A. McAllesterl. Pac-bayesian model averaging. In Proceedings of the eleventh annual conference on Computational learning theory, pages 164- 170, 1999.","title":"Pac-bayesian model averaging","authors":"David A Mcallesterl","year":"1999","venue":"Proceedings of the eleventh annual conference on Computational learning theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McAllesterl, 2003] David A. McAllesterl. Simplified pac- bayesian margin bounds. Lecture notes in computer sci- ence, pages 203-215, 2003.","title":"Simplified pacbayesian margin bounds","authors":"David A Mcallesterl","year":"2003","venue":"Lecture notes in computer science","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Neyshabur et al., 2017] Behnam Neyshabur, Srinadh Bho- janapalli, David McAllester, and Nathan Srebro. Ex- ploring generalization in deep learning. arXiv preprint arXiv:1706.08947, 2017.","title":"Exploring generalization in deep learning","authors":"Srinadh Behnam Neyshabur, David Bhojanapalli, Nathan Mcallester, Srebro","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pierre-Antoine and Gallivan, 2006] Absil Pierre-Antoine and Kyle Gallivan. Joint diagonalization on the oblique manifold for independent component analysis. ICASSP, 2006.","title":"Joint diagonalization on the oblique manifold for independent component analysis","authors":"Pierre-Antoine, Gallivan ; Absil Pierre-Antoine, Kyle Gallivan","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pierre-Antoine et al., 2009] Absil Pierre-Antoine, Robert. Mahony, and Rodolphe Sepulchre. Optimization algo- rithms on matrix manifolds. Princeton Press, 2009.","title":"Optimization algorithms on matrix manifolds","authors":"Pierre-Antoine","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2018] Huan Wang, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Identifying gen- eralization properties in neural network. arXiv preprint arXiv:1809.07402, 2018.","title":"Identifying generalization properties in neural network","authors":"Huan Wang, Nitish Shirish Keskar, Caiming Xiong, Richard Socher","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wen et al., 2018] Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chunpeng Wu, Yiran Chen, and Hai Lii. Smoothout: Smoothing out sharp minima to improve generalization in deep learning. arXiv preprint arXiv:1805.07898, 2018.","title":"Smoothout: Smoothing out sharp minima to improve generalization in deep learning","authors":"Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chunpeng Wu, Yiran Chen, Hai Lii","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2018] Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural In- formation Processing Systems 31 (NeurlPS 2018), 2018.","title":"How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective","authors":"Lei Wu, Chao Ma, Weinan E","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan et al., 2018] Xiaoyong Yuan, Zheng Feng, Matthew Norton, and Xiaolin Lii. Generalized batch normalization: Towards accelerating deep neural networks. arXiv preprint arXiv:1812.03271, 2018.","title":"Generalized batch normalization: Towards accelerating deep neural networks","authors":"Xiaoyong Yuan, Zheng Feng, Matthew Norton, Xiaolin Lii","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2016] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In ICLR'2017 arXiv:1611.03530, 2016.","title":"Understanding deep learning requires rethinking generalization","authors":"Zhang","year":"2016","venue":"ICLR'2017","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}