{"type":"ScholarlyArticle","title":"Ridgeless Regression with Random Features","authors":[{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Yong Liu","slug":"yong-liu","given":"Yong","family":"Liu","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China"},{"name":"Yingying Zhang","slug":"yingying-zhang","given":"Yingying","family":"Zhang","affiliation":"School of Mathematics and Information Sciences , Yantai University"}],"abstract":"Recent theoretical studies illustrated that kernel ridgeless regression can guarantee good generalization ability without an explicit regularization. In this paper, we investigate the statistical properties of ridgeless regression with random features and stochastic gradient descent. We explore the effect of factors in the stochastic gradient and random features, respectively. Specifically, random features error exhibits the double-descent curve. Motivated by the theoretical findings, we propose a tunable kernel algorithm that optimizes the spectral density of kernel during training. Our work bridges the interpolation theory and practical algorithm.","doi":"10.24963/ijcai.2022/445","url":"https://ijcai.org/proceedings/2022/445","pdf":"https://ijcai.org/proceedings/2022/0445.pdf","markdown":"https://ijcai.org/proceedings/2022/445.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/445","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":445,"pages":{"first":3208,"last":3214},"keywords":["Machine Learning: Learning Theory","Machine Learning: Kernel Methods"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986927","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985070","length_sec":349,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Advani et al., 2020] Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gen- eralization error in neural networks. Neural Networks, 132:428-446, 2020.","title":"High-dimensional dynamics of generalization error in neural networks","authors":"Andrew M Madhu S Advani, Haim Saxe, Sompolinsky","year":"2020","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett et al., 2005] Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local Rademacher complexities. The Annals of Statistics, 33(4):1497-1537, 2005.","title":"Local Rademacher complexities","authors":"L Peter, Olivier Bartlett, Shahar Bousquet, Mendelson","year":"2005","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartlett et al., 2020] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in lin- ear regression. Proceedings of the National Academy of Sciences, 117(48):30063-30070, 2020.","title":"Benign overfitting in linear regression","authors":"Philip M Peter L Bartlett, Gábor Long, Alexander Lugosi, Tsigler","year":"2020","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Belkin et al., 2018] Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. arXiv preprint arXiv:1802.01396, 2018.","title":"To understand deep learning we need to understand kernel learning","authors":"Mikhail Belkin, Siyuan Ma, Soumik Mandal","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caponnetto and De Vito, 2007] Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-368, 2007.","title":"Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm","authors":"De Caponnetto, Vito","year":"2007","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carratino et al., 2018] Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with sgd and random fea- tures. In Advances in Neural Information Processing Sys- tems 31 (NeurIPS), pages 10192-10203, 2018.","title":"Learning with sgd and random features","authors":"Carratino","year":"2018","venue":"Advances in Neural Information Processing Systems 31 (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geiger et al., 2020] Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scal- ing description of generalization with number of parame- ters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2):023401, 2020.","title":"Scaling description of generalization with number of parameters in deep learning","authors":"Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Giulio Stéphane D'ascoli, Clément Biroli, Matthieu Hongler, Wyart","year":"2020","venue":"Journal of Statistical Mechanics: Theory and Experiment","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghorbani et al., 2021] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Lin- earized two-layers neural networks in high dimension. The Annals of Statistics, 49(2):1029-1054, 2021.","title":"Linearized two-layers neural networks in high dimension","authors":"Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, Andrea Montanari","year":"2021","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hastie et al., 2019] Trevor Hastie, Andrea Montanari, Sa- haron Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560, 2019.","title":"Surprises in highdimensional ridgeless least squares interpolation","authors":"Trevor Hastie, Andrea Montanari, Saharon Rosset, Ryan J Tibshirani","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jacot et al., 2020] Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Im- plicit regularization of random feature models. In Inter- national Conference on Machine Learning, pages 4631- 4640. PMLR, 2020.","title":"Implicit regularization of random feature models","authors":"Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, Franck Gabriel","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Liu, 2022] Jian Li and Yong Liu. Optimal rates for distributed learning with random features. In Proceedings of the 31st International Joint Conference on Artificial In- telligence (IJCAI), 2022.","title":"Optimal rates for distributed learning with random features","authors":"Jian Li, Yong Liu","year":"2022","venue":"Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, and Weiping Wang. Multi-class learning: From theory to algorithm. In Advances in Neural Informa- tion Processing Systems 31, pages 1591-1600, 2018.","title":"Multi-class learning: From theory to algorithm","authors":"Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, Weiping Wang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Jian Li, Yong Liu, Rong Yin, and Weiping Wang. Multi-class learning using unlabeled samples : Theory and algorithm. In Proceedings of the 28th Interna- tional Joint Conference on Artificial Intelligence (IJCAI), 2019.","title":"Multi-class learning using unlabeled samples : Theory and algorithm","authors":"Jian Li, Yong Liu, Rong Yin, Weiping Wang","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2019/399"},{"position":14,"raw":"[Li et al., 2020] Jian Li, Yong Liu, and Weiping Wang. Au- tomated spectral kernel learning. In Thirty-Four AAAI Conference on Artificial Intelligence, 2020.","title":"Automated spectral kernel learning","authors":"Jian Li, Yong Liu, Weiping Wang","year":"2020","venue":"Thirty-Four AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang and Rakhlin, 2020] Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel \"ridgeless\" regression can generalize. The Annals of Statistics, 48(3):1329-1347, 2020.","title":"Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel \"ridgeless\" regression can generalize","authors":"Rakhlin Liang","year":"2020","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mei and Montanari, 2019] Song Mei and Andrea Monta- nari. The generalization error of random features regres- sion: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 2019.","title":"Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve","authors":"Montanari Mei","year":"2019","venue":"Communications on Pure and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nakkiran et al., 2021] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.","title":"Deep double descent: Where bigger models and more data hurt","authors":"Nakkiran","year":"2021","venue":"Journal of Statistical Mechanics: Theory and Experiment","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 21 (NIPS), pages 1177-1184, 2007.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2007","venue":"Advances in Neural Information Processing Systems 21 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rahimi and Recht, 2008] Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing min- imization with randomization in learning. In Advances in Neural Information Processing Systems 22 (NIPS), pages 1313-1320, 2008.","title":"Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2008","venue":"Advances in Neural Information Processing Systems 22 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rudi and Rosasco, 2017] Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with ran- dom features. In Advances in Neural Information Process- ing Systems 30 (NIPS), pages 3215-3225, 2017.","title":"Generalization properties of learning with random features","authors":"Rosasco Rudi, Alessandro Rudi, Lorenzo Rosasco","year":"2017","venue":"Advances in Neural Information Processing Systems 30 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rudi et al., 2018] Alessandro Rudi, Daniele Calandriello, Luigi Carratino, and Lorenzo Rosasco. On fast lever- age score sampling and optimal learning. In Advances in Neural Information Processing Systems, pages 5672- 5682, 2018.","title":"On fast leverage score sampling and optimal learning","authors":"Alessandro Rudi, Daniele Calandriello, Luigi Carratino, Lorenzo Rosasco","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Smale and Zhou, 2007] Steve Smale and Ding-Xuan Zhou. Learning theory estimates via integral operators and their approximations. Constructive approximation, 26(2):153- 172, 2007.","title":"Learning theory estimates via integral operators and their approximations","authors":"Zhou ; ] Steve Smale, Ding-Xuan Smale, Zhou","year":"2007","venue":"Constructive approximation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.","title":"Understanding deep learning (still) requires rethinking generalization","authors":"Zhang","year":"2021","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}