{"type":"ScholarlyArticle","title":"Towards Sharp Analysis for Distributed Learning with Random Features","authors":[{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Yong Liu","slug":"yong-liu","given":"Yong","family":"Liu","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China"}],"abstract":"In recent studies, the generalization properties for distributed learning and random features assumed the existence of the target concept over the hypothesis space. However, this strict condition is not applicable to the more common non-attainable case. In this paper, using refined proof techniques, we first extend the optimal rates for distributed learning with random features to the non-attainable case. Then, we reduce the number of required random features via data-dependent generating strategy, and improve the allowed number of partitions with additional unlabeled data. Theoretical analysis shows these techniques remarkably reduce computational cost while preserving the optimal generalization accuracy under standard assumptions. Finally, we conduct several experiments on both simulated and real-world datasets, and the empirical results validate our theoretical findings.","doi":"10.24963/ijcai.2023/436","url":"https://ijcai.org/proceedings/2023/436","pdf":"https://ijcai.org/proceedings/2023/0436.pdf","markdown":"https://ijcai.org/proceedings/2023/436.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/436","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":436,"pages":{"first":3920,"last":3928},"keywords":["Machine Learning: ML: Learning theory","Machine Learning: ML: Kernel methods"],"videos":[],"references":[{"position":1,"raw":"[Alaoui and Mahoney, 2015] Ahmed Alaoui and Michael W Mahoney. Fast randomized kernel ridge regression with statistical guarantees. In Advances in Neural Information Processing Systems 28 (NIPS), pages 775-783, 2015.","title":"Fast randomized kernel ridge regression with statistical guarantees","authors":"Ahmed Alaoui, Michael W Mahoney","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avron et al., 2017] Haim Avron, Kenneth L Clarkson, and David P Woodruff. Faster kernel ridge regression using sketching and preconditioning. SIAM Journal on Matrix Analysis and Applications, 38(4):1116-1138, 2017.","title":"Faster kernel ridge regression using sketching and preconditioning","authors":"Kenneth L Haim Avron, David P Clarkson, Woodruff","year":"2017","venue":"SIAM Journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bach, 2013] Francis Bach. Sharp analysis of low-rank ker- nel matrix approximations. In Conference on Learning Theory, pages 185-209, 2013.","title":"Sharp analysis of low-rank kernel matrix approximations","authors":"Francis Bach","year":"2013","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bach, 2017] Francis Bach. On the equivalence between ker- nel quadrature rules and random feature expansions. Jour- nal of Machine Learning Research, 18(21):1-38, 2017. 00054.","title":"On the equivalence between kernel quadrature rules and random feature expansions","authors":"Francis Bach","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bottou and Bousquet, 2008] Léon Bottou and Olivier Bous- quet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems 21 (NIPS), pages 161-168, 2008.","title":"The tradeoffs of large scale learning","authors":"Bousquet ; Bottou, Olivier Bousquet","year":"2008","venue":"Advances in Neural Information Processing Systems 21 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Caponnetto and De Vito, 2007] Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-368, 2007.","title":"Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm","authors":"De Caponnetto, Vito","year":"2007","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Carratino et al., 2018] Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with sgd and random fea- tures. In Advances in Neural Information Processing Sys- tems 31 (NeurIPS), pages 10192-10203, 2018.","title":"Learning with sgd and random features","authors":"Carratino","year":"2018","venue":"Advances in Neural Information Processing Systems 31 (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chang et al., 2017] Xiangyu Chang, Shao-Bo Lin, and Ding-Xuan Zhou. Distributed semi-supervised learning with kernel ridge regression. Journal of Machine Learning Research, 18(1):1493-1514, 2017.","title":"Distributed semi-supervised learning with kernel ridge regression","authors":"Xiangyu Chang, Shao-Bo Lin, Ding-Xuan Zhou","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2017] Zheng-Chu Guo, Shao-Bo Lin, and Ding- Xuan Zhou. Learning theory of distributed spectral algo- rithms. Inverse Problems, 33(7):074009, 2017.","title":"Learning theory of distributed spectral algorithms","authors":"Zheng-Chu Guo, Shao-Bo Lin, Ding-Xuan Zhou","year":"2017","venue":"Inverse Problems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kriukova et al., 2017] Galyna Kriukova, Sergiy Pereverzyev, and Pavlo Tkachenko. Nyström type subsampling analyzed as a regularized projection. Inverse Problems, 33(7):074001, 2017.","title":"Nyström type subsampling analyzed as a regularized projection","authors":"Galyna Kriukova, Sergiy Pereverzyev, Pavlo Tkachenko","year":"2017","venue":"Inverse Problems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2018] Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, and Weiping Wang. Multi-class learning: From theory to algorithm. In Advances in Neural Informa- tion Processing Systems 31, pages 1591-1600, 2018.","title":"Multi-class learning: From theory to algorithm","authors":"Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, Weiping Wang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019a] Jian Li, Yong Liu, Rong Yin, and Weip- ing Wang. Approximate manifold regularization: Scalable algorithm and generalization analysis. In IJCAI, pages 2887-2893, 2019.","title":"Approximate manifold regularization: Scalable algorithm and generalization analysis","authors":"Li","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/400"},{"position":13,"raw":"[Li et al., 2019b] Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards a unified analysis of ran- dom fourier features. In International Conference on Ma- chine Learning, pages 3905-3914. PMLR, 2019.","title":"Towards a unified analysis of random fourier features","authors":"Li","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Jian Li, Yong Liu, and Weiping Wang. Au- tomated spectral kernel learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4618- 4625, 2020.","title":"Automated spectral kernel learning","authors":"Jian Li, Yong Liu, Weiping Wang","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li, 2021] Zhu Li. Sharp analysis of random fourier features in classification. arXiv preprint arXiv:2109.10623, 2021.","title":"Sharp analysis of random fourier features in classification","authors":"Zhu Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin and Cevher, 2018] Junhong Lin and Volkan Cevher. Optimal distributed learning with multi-pass stochastic gradient methods. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 3098- 3107, 2018.","title":"Optimal distributed learning with multi-pass stochastic gradient methods","authors":"Junhong Lin, Volkan Cevher","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin and Cevher, 2020] Junhong Lin and Volkan Cevher. Optimal convergence for distributed learning with stochas- tic gradient methods and spectral algorithms. Journal of Machine Learning Research, 21(147):1-63, 2020.","title":"Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms","authors":"Junhong Lin, Volkan Cevher","year":"2020","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin and Rosasco, 2016] Junhong Lin and Lorenzo Rosasco. Optimal learning for multi-pass stochastic gradient meth- ods. In Advances in Neural Information Processing Sys- tems 29 (NIPS), pages 4556-4564, 2016.","title":"Optimal learning for multi-pass stochastic gradient methods","authors":"Lin, Rosasco ;, Junhong Lin, Lorenzo Rosasco","year":"2016","venue":"Advances in Neural Information Processing Systems 29 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2017] Shao-Bo Lin, Xin Guo, and Ding-Xuan Zhou. Distributed learning with regularized least squares. The Journal of Machine Learning Research, 18(1):3202- 3232, 2017.","title":"Distributed learning with regularized least squares","authors":"Xin Shao-Bo Lin, Ding-Xuan Guo, Zhou","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2020] Shao-Bo Lin, Di Wang, and Ding-Xuan Zhou. Distributed kernel ridge regression with communi- cations. Journal of Machine Learning Research, 21(93):1- 38, 2020.","title":"Distributed kernel ridge regression with communications","authors":"Di Shao-Bo Lin, Ding-Xuan Wang, Zhou","year":"2020","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2021] Yong Liu, Jiankun Liu, and Shuqiang Wang. Effective distributed learning with random features: Improved bounds and algorithms. In International Confer- ence on Learning Representations, 2021.","title":"Effective distributed learning with random features: Improved bounds and algorithms","authors":"Yong Liu, Jiankun Liu, Shuqiang Wang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mücke and Blanchard, 2018] Nicole Mücke and Gilles Blanchard. Parallelizing spectrally regularized kernel al- gorithms. The Journal of Machine Learning Research, 19(1):1069-1097, 2018.","title":"Parallelizing spectrally regularized kernel algorithms","authors":"Nicole Mücke, Gilles Blanchard","year":"2018","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 21 (NIPS), pages 1177-1184, 2007.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2007","venue":"Advances in Neural Information Processing Systems 21 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rudi and Rosasco, 2017] Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with ran- dom features. In Advances in Neural Information Process- ing Systems 30 (NIPS), pages 3215-3225, 2017.","title":"Generalization properties of learning with random features","authors":"Rosasco Rudi, Alessandro Rudi, Lorenzo Rosasco","year":"2017","venue":"Advances in Neural Information Processing Systems 30 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rudi et al., 2015] Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computa- tional regularization. In Advances in Neural Information Processing Systems 28 (NIPS), pages 1657-1665, 2015.","title":"Less is more: Nyström computational regularization","authors":"Alessandro Rudi, Raffaello Camoriano, Lorenzo Rosasco","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rudi et al., 2018] Alessandro Rudi, Daniele Calandriello, Luigi Carratino, and Lorenzo Rosasco. On fast leverage score sampling and optimal learning. In Advances in Neu- ral Information Processing Systems 31 (NeurIPS), pages 5672-5682, 2018.","title":"On fast leverage score sampling and optimal learning","authors":"Alessandro Rudi, Daniele Calandriello, Luigi Carratino, Lorenzo Rosasco","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Smale and Zhou, 2007] Steve Smale and Ding-Xuan Zhou. Learning theory estimates via integral operators and their approximations. Constructive approximation, 26(2):153- 172, 2007.","title":"Learning theory estimates via integral operators and their approximations","authors":"Zhou ; ] Steve Smale, Ding-Xuan Smale, Zhou","year":"2007","venue":"Constructive approximation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Steinwart and Christmann, 2008] Ingo Steinwart and An- dreas Christmann. Support vector machines. Springer Ver- lag, New York, 2008.","title":"Ingo Steinwart and Andreas Christmann. Support vector machines","authors":"Christmann Steinwart","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sun and Wu, 2020] Hongwei Sun and Qiang Wu. Opti- mal rates of distributed regression with imperfect kernels. arXiv preprint arXiv:2006.16744, 2020.","title":"Optimal rates of distributed regression with imperfect kernels","authors":"Hongwei Sun, Qiang Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2018] Yitong Sun, Anna Gilbert, and Ambuj Tewari. But how does it work in theory? linear svm with random features. In Advances in Neural Information Pro- cessing Systems, pages 3379-3388, 2018.","title":"But how does it work in theory? linear svm with random features","authors":"Yitong Sun, Anna Gilbert, Ambuj Tewari","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Williams and Seeger, 2001] Christopher KI Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In Advances in Neural Information Pro- cessing Systems 14 (NIPS), pages 682-688, 2001.","title":"Using the nyström method to speed up kernel machines","authors":"K I Christopher, Matthias Williams, Seeger","year":"2001","venue":"Advances in Neural Information Processing Systems 14 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2015] Yuchen Zhang, John Duchi, and Mar- tin Wainwright. Divide and conquer kernel ridge regres- sion: A distributed algorithm with minimax optimal rates. Journal of Machine Learning Research, 16(1):3299-3340, 2015.","title":"Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates","authors":"Yuchen Zhang, John Duchi, Martin Wainwright","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}