{"type":"ScholarlyArticle","title":"Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Rényi's Entropy Perspective","authors":[{"name":"Yuxin Dong","slug":"yuxin-dong","given":"Yuxin","family":"Dong","affiliation":"School of Computer Science and Technology , Xi'an Jiaotong University , Xi'an 710049 , China"},{"name":"Tieliang Gong","slug":"tieliang-gong","given":"Tieliang","family":"Gong","affiliation":"School of Computer Science and Technology , Xi'an Jiaotong University , Xi'an 710049 , China"},{"name":"Hong Chen","slug":"hong-chen","given":"Hong","family":"Chen","affiliation":"College of Science , Huazhong Agriculture University , Wuhan 430070 , China"},{"name":"Chen Li","slug":"chen-li","given":"Chen","family":"Li","affiliation":"School of Computer Science and Technology , Xi'an Jiaotong University , Xi'an 710049 , China"}],"abstract":"Recently, information-theoretic analysis has become a popular framework for understanding the generalization behavior of deep neural networks. It allows a direct analysis for stochastic gradient / Langevin descent (SGD/SGLD) learning algorithms without strong assumptions such as Lipschitz or convexity conditions. However, the current generalization error bounds within this framework are still far from optimal, while substantial improvements on these bounds are quite challenging due to the intractability of high-dimensional information quantities. To address this issue,  we first propose a novel information theoretical measure: kernelized Rényi's entropy, by utilizing operator representation in Hilbert space. It inherits the properties of Shannon's entropy and can be effectively calculated via simple random sampling, while remaining independent of the input dimension. We then establish the generalization error bounds for SGD/SGLD under kernelized Rényi's entropy, where the mutual information quantities can be directly calculated, enabling evaluation of the tightness of each intermediate step. We show that our information-theoretical bounds depend on the statistics of the stochastic gradients evaluated along with the iterates, and are rigorously tighter than the current state-of-the-art (SOTA) results. The theoretical findings are also supported by large-scale empirical studies.","doi":"10.24963/ijcai.2023/405","url":"https://ijcai.org/proceedings/2023/405","pdf":"https://ijcai.org/proceedings/2023/0405.pdf","markdown":"https://ijcai.org/proceedings/2023/405.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/405","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":405,"pages":{"first":3642,"last":3650},"keywords":["Machine Learning: ML: Learning theory","Machine Learning: ML: Theory of deep learning"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2018] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 254-263. PMLR, 2018.","title":"Stronger generalization bounds for deep nets via a compression approach","authors":"Sanjeev Arora, Rong Ge, Behnam Neyshabur, Yi Zhang","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett et al., 2017] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information pro- cessing systems, 30, 2017.","title":"Spectrally-normalized margin bounds for neural networks","authors":"Dylan J Peter L Bartlett, Matus J Foster, Telgarsky","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bassily et al., 2020] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33:4381-4391, 2020.","title":"Stability of stochastic gradient descent on nonsmooth convex losses","authors":"Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, Kunal Talwar","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bottou et al., 2018] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.","title":"Optimization methods for large-scale machine learning","authors":"Bottou","year":"2018","venue":"Siam Review","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Camuto et al., 2021] Alexander Camuto, Xiaoyu Wang, Lingjiong Zhu, Chris Holmes, Mert Gurbuzbalaban, and Umut Simsekli. Asymmetric heavy tails and implicit bias in gaussian noise injections. In International Conference on Machine Learning, pages 1249-1260. PMLR, 2021.","title":"Asymmetric heavy tails and implicit bias in gaussian noise injections","authors":"Alexander Camuto, Xiaoyu Wang, Lingjiong Zhu, Chris Holmes, Mert Gurbuzbalaban, Umut Simsekli","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dangel et al., 2020] Felix Dangel, Frederik Kunstner, and Philipp Hennig. Backpack: Packing more into backprop. In International Conference on Learning Representations, 2020.","title":"Backpack: Packing more into backprop","authors":"Felix Dangel, Frederik Kunstner, Philipp Hennig","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Giraldo et al., 2014] Luis Gonzalo Sanchez Giraldo, Murali Rao, and Jose C Principe. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory, 61(1):535-548, 2014.","title":"Measures of entropy from data using infinitely divisible kernels","authors":"Luis Gonzalo, Sanchez Giraldo, Murali Rao, Jose C Principe","year":"2014","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gurbuzbalaban et al., 2021] Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pages 3964-3975. PMLR, 2021.","title":"The heavy-tail phenomenon in sgd","authors":"Mert Gurbuzbalaban, Umut Simsekli, Lingjiong Zhu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hardt et al., 2016] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochas- tic gradient descent. In International conference on ma- chine learning, pages 1225-1234. PMLR, 2016.","title":"Train faster, generalize better: Stability of stochastic gradient descent","authors":"Moritz Hardt, Ben Recht, Yoram Singer","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lei et al., 2021a] Yunwen Lei, Ting Hu, and Ke Tang. Gen- eralization performance of multi-pass stochastic gradient descent with convex loss functions. Journal of Machine Learning Research, 22:1-41, 2021.","title":"Generalization performance of multi-pass stochastic gradient descent with convex loss functions","authors":"Lei","year":"2021","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lei et al., 2021b] Yunwen Lei, Zhenhuan Yang, Tianbao Yang, and Yiming Ying. Stability and generalization of stochastic gradient methods for minimax problems. In International Conference on Machine Learning, pages 6175-6186. PMLR, 2021.","title":"Stability and generalization of stochastic gradient methods for minimax problems","authors":"Lei","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Liu, 2022] Shaojie Li and Yong Liu. High probabil- ity guarantees for nonconvex stochastic gradient descent with heavy tails. In International Conference on Machine Learning, pages 12931-12963. PMLR, 2022.","title":"High probability guarantees for nonconvex stochastic gradient descent with heavy tails","authors":"Li, Liu ; ] Shaojie Li, Yong Liu","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mahoney and Martin, 2019] Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning, pages 4284-4293. PMLR, 2019.","title":"Traditional and heavy tailed self regularization in neural network models","authors":"Martin ; Mahoney, Michael Mahoney, Charles Martin","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Martin and Mahoney, 2021] Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. J. Mach. Learn. Res., 22(165):1-73, 2021.","title":"Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning","authors":"H Charles, Michael W Martin, Mahoney","year":"2021","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Negrea et al., 2019] Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems, 32, 2019.","title":"Information-theoretic generalization bounds for sgld via data-dependent estimates","authors":"Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, Daniel M Roy","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Neu et al., 2021] Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M Roy. Information- theoretic generalization bounds for stochastic gradient de- scent. In Conference on Learning Theory, pages 3526- 3545. PMLR, 2021.","title":"Informationtheoretic generalization bounds for stochastic gradient descent","authors":"Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, Daniel M Roy","year":"2021","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Neyshabur et al., 2018] Behnam Neyshabur, Srinadh Bho- janapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations, 2018.","title":"A pac-bayesian approach to spectrally-normalized margin bounds for neural networks","authors":"Srinadh Behnam Neyshabur, Nathan Bhojanapalli, Srebro","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pensia et al., 2018] Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization error bounds for noisy, iterative algo- rithms. In 2018 IEEE International Symposium on Infor- mation Theory (ISIT), pages 546-550. IEEE, 2018.","title":"Generalization error bounds for noisy, iterative algorithms","authors":"Ankit Pensia, Varun Jog, Po-Ling Loh","year":"2018","venue":"IEEE International Symposium on Information Theory (ISIT)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang and Mao, 2021] Ziqiao Wang and Yongyi Mao. On the generalization of models trained with sgd: Information-theoretic bounds and implications. In Inter- national Conference on Learning Representations, 2021.","title":"On the generalization of models trained with sgd: Information-theoretic bounds and implications","authors":"Wang, Mao ; ] Ziqiao Wang, Yongyi Mao","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021] Hao Wang, Yizhe Huang, Rui Gao, and Flavio Calmon. Analyzing the generalization capability of sgld using properties of gaussian channels. Advances in Neural Information Processing Systems, 34:24222-24234, 2021.","title":"Analyzing the generalization capability of sgld using properties of gaussian channels","authors":"Hao Wang, Yizhe Huang, Rui Gao, Flavio Calmon","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu and Raginsky, 2017] Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, 30, 2017.","title":"Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms","authors":"Raginsky Xu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2019] Jun Yang, Shengyang Sun, and Daniel M Roy. Fast-rate pac-bayes generalization bounds via shifted rademacher processes. Advances in Neural Information Processing Systems, 32, 2019.","title":"Fast-rate pac-bayes generalization bounds via shifted rademacher processes","authors":"Yang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2021a] Zhenhuan Yang, Yunwen Lei, Siwei Lyu, and Yiming Ying. Stability and differential privacy of stochastic gradient descent for pairwise learning with non-smooth loss. In International Conference on Artifi- cial Intelligence and Statistics, pages 2026-2034. PMLR, 2021.","title":"Stability and differential privacy of stochastic gradient descent for pairwise learning with non-smooth loss","authors":"Yang","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2021b] Zhenhuan Yang, Yunwen Lei, Puyu Wang, Tianbao Yang, and Yiming Ying. Simple stochastic and online gradient descent algorithms for pairwise learn- ing. Advances in Neural Information Processing Systems, 34:20160-20171, 2021.","title":"Simple stochastic and online gradient descent algorithms for pairwise learning","authors":"Yang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2019] Shujian Yu, Luis Gonzalo Sanchez Giraldo, Robert Jenssen, and Jose C Principe. Multivariate exten- sion of matrix-based rényi's α-order entropy functional. IEEE transactions on pattern analysis and machine intel- ligence, 42(11):2960-2966, 2019.","title":"Multivariate extension of matrix-based rényi's α-order entropy functional","authors":"Yu","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2021] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.","title":"Understanding deep learning (still) requires rethinking generalization","authors":"Zhang","year":"2021","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2018] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generalization bounds at the imagenet scale: a pac- bayesian compression approach. In Advances in Neural Information Processing Systems, 2018.","title":"Non-vacuous generalization bounds at the imagenet scale: a pacbayesian compression approach","authors":"Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, Peter Orbanz","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}