{"type":"ScholarlyArticle","title":"Efficient Kernel Selection via Spectral Analysis","authors":[{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Yong Liu","slug":"yong-liu","given":"Yong","family":"Liu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; Tianjin University"},{"name":"Hailun Lin","slug":"hailun-lin","given":"Hailun","family":"Lin","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Yinliang Yue","slug":"yinliang-yue","given":"Yinliang","family":"Yue","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Weiping Wang","slug":"weiping-wang","given":"Weiping","family":"Wang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"}],"abstract":"Kernel selection is a fundamental problem of kernel methods. Existing measures for kernel selection either provide less theoretical guarantee or have high computational complexity. In this paper, we propose a novel kernel selection criterion based on a newly defined spectral measure of a kernel matrix, with sound theoretical foundation and high computational efficiency. We first show that the spectral measure can be used to derive generalization bounds for some kernel-based algorithms. By minimizing the derived generalization bounds, we propose the kernel selection criterion with spectral measure. Moreover, we demonstrate that the popular minimum graph cut and maximum mean discrepancy are two special cases of the proposed criterion. Experimental results on lots of data sets show that our proposed criterion can not only give the comparable results as the state-of-the-art criterion, but also significantly improve the efficiency.","doi":"10.24963/ijcai.2017/295","url":"https://ijcai.org/proceedings/2017/295","pdf":"https://ijcai.org/proceedings/2017/0295.pdf","markdown":"https://ijcai.org/proceedings/2017/295.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/295","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":295,"pages":{"first":2124,"last":2130},"keywords":["Machine Learning: Classification","Machine Learning: Data Mining","Machine Learning: Kernel Methods"],"videos":[],"references":[{"position":1,"raw":"[Bartlett and Mendelson, 2002] Peter L. Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: risk bounds and structural results. Journal of Machine Learn- ing Research, 3:463-482, 2002.","title":"Rademacher and Gaussian complexities: risk bounds and structural results","authors":"Peter L Bartlett, Shahar Mendelson","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett et al., 2005] Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local Rademacher complexities. The Annals of Statistics, 33(4):1497-1537, 2005.","title":"Local Rademacher complexities","authors":"L Peter, Olivier Bartlett, Shahar Bousquet, Mendelson","year":"2005","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bousquet and Elisseeff, 2002] Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.","title":"Stability and generalization","authors":"Elisseeff Bousquet, Olivier Bousquet, André Elisseeff","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cawley, 2006] Gavin C. Cawley. Leave-one-out cross- validation based model selection criteria for weighted LS- SVMs. In Proceeding of the International Joint Con- ference on Neural Networks (IJCNN 2006), pages 1661- 1668, 2006.","title":"Leave-one-out crossvalidation based model selection criteria for weighted LS-SVMs","authors":"C Gavin, Cawley","year":"2006","venue":"Proceeding of the International Joint Conference on Neural Networks (IJCNN 2006)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chapelle et al., 2002] Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, and Sayan Mukherjee. Choosing mul- tiple parameters for support vector machines. Machine Learning, 46(1-3):131-159, 2002.","title":"Choosing multiple parameters for support vector machines","authors":"Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, Sayan Mukherjee","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cortes et al., 2010] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Two-stage learning kernel algo- rithms. In Proceedings of the 27th Conference on Machine Learning (ICML 2010), pages 239-246, 2010.","title":"Two-stage learning kernel algorithms","authors":"Corinna Cortes, Mehryar Mohri, Afshin Rostamizadeh","year":"2010","venue":"Proceedings of the 27th Conference on Machine Learning (ICML 2010)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cortes et al., 2013] Corinna Cortes, Marius Kloft, and Mehryar Mohri. Learning kernels using local Rademacher complexity. In Advances in Neural Information Process- ing Systems 25 (NIPS 2013), pages 2760-2768. MIT Press, 2013.","title":"Learning kernels using local Rademacher complexity","authors":"Corinna Cortes, Marius Kloft, Mehryar Mohri","year":"2013","venue":"Advances in Neural Information Processing Systems 25 (NIPS 2013)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cristianini et al., 2001] Nello Cristianini, John Shawe- Taylor, André Elisseeff, and Jaz S. Kandola. On kernel- target alignment. In Advances in Neural Information Pro- cessing Systems 14 (NIPS 2001), pages 367-373, 2001.","title":"On kerneltarget alignment","authors":"Nello Cristianini, John Shawe-Taylor, André Elisseeff, Jaz S Kandola","year":"2001","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Debruyne et al., 2008] Michiel Debruyne, Mia Hubert, and Johan A.K. Suykens. Model selection in kernel based re- gression using the influence function. Journal of Machine Learning Research, 9:2377-2400, 2008.","title":"Model selection in kernel based regression using the influence function","authors":"Michiel Debruyne, Mia Hubert, Johan A K Suykens","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao and Zhou, 2013] Wei Gao and Zhi-Hua Zhou. On the doubt about margin explanation of boosting. Artificial In- telligence, 203:1-18, 2013.","title":"On the doubt about margin explanation of boosting","authors":"Wei Gao, Zhi-Hua Zhou","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Golub et al., 1979] Gene H. Golub, Michael Heath, and Grace Wahba. Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics, 21(2):215-223, 1979.","title":"Generalized cross-validation as a method for choosing a good ridge parameter","authors":"Gene H Golub, Michael Heath, Grace Wahba","year":"1979","venue":"Technometrics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gretton et al., 2007] Arthur Gretton, Karsten M. Borg- wardt, Malte Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel method for the twosample problem. In Advances in Neural Information Processing Systems 15 (NIPS 2007), pages 513-520, 2007.","title":"A kernel method for the twosample problem","authors":"Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, Alexander J Smola","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kloft and Blanchard, 2011] Marius Kloft and Gilles Blan- chard. The local Rademacher complexity of lp-norm mul- tiple kernel learning. In Advances in Neural Information Processing Systems 23 (NIPS 2011), pages 2438-2446.","title":"Marius Kloft and Gilles Blanchard. The local Rademacher complexity of lp-norm multiple kernel learning","authors":"Blanchard Kloft","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kloft et al., 2011] Marius Kloft, Ulf Brefeld, Sören Son- nenburg, and Alexander Zien. Lp-norm multiple kernel learning. Journal of Machine Learning Research, 12:953- 997, 2011.","title":"Lp-norm multiple kernel learning","authors":"Marius Kloft, Ulf Brefeld, Sören Sonnenburg, Alexander Zien","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lanckriet et al., 2004] Gert R. G. Lanckriet, Nello Cristian- ini, Peter L. Bartlett, Laurent El Ghaoui, and Michael I. Jordan. Learning the kernel matrix with semidefinite pro- gramming. Journal of Machine Learning Research, 5:27- 72, 2004.","title":"Learning the kernel matrix with semidefinite programming","authors":"R G Gert, Nello Lanckriet, Peter L Cristianini, Laurent El Bartlett, Michael I Ghaoui, Jordan","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu and Liao, 2014] Yong Liu and Shizhong Liao. Prevent- ing over-fitting of cross-validation with kernel stability. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Dis- covery in Databases (ECML), pages 290-305, 2014.","title":"Preventing over-fitting of cross-validation with kernel stability","authors":"Yong Liu, Shizhong Liao","year":"2014","venue":"Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu and Liao, 2015] Yong Liu and Shizhong Liao. Eigen- values ratio for kernel selection of kernel methods. In Pro- ceedings of the Twenty-Ninth AAAI Conference on Artifi- cial Intelligence (AAAI), pages 2814-2820, 2015.","title":"Eigenvalues ratio for kernel selection of kernel methods","authors":"Yong Liu, Shizhong Liao","year":"2015","venue":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu and Liao, 2017] Yong Liu and Shizhong Liao. Granu- larity selection for cross-validation of svm. Information Sciences, 378:475-483, 2017.","title":"Granularity selection for cross-validation of svm","authors":"Yong Liu, Shizhong Liao","year":"2017","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2013] Yong Liu, Shali Jiang, and Shizhong Liao. Eigenvalues perturbation of integral operator for kernel s- election. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013), pages 2189-2198, 2013.","title":"Eigenvalues perturbation of integral operator for kernel selection","authors":"Yong Liu, Shali Jiang, Shizhong Liao","year":"2013","venue":"Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2014] Yong Liu, Shali Jiang, and Shizhong Liao. Efficient approximation of cross-validation for kernel methods using Bouligand influence function. In Proceed- ings of The 31st International Conference on Machine Learning (ICML 2014 (1)), pages 324-332, 2014.","title":"Efficient approximation of cross-validation for kernel methods using Bouligand influence function","authors":"Yong Liu, Shali Jiang, Shizhong Liao","year":"2014","venue":"Proceedings of The 31st International Conference on Machine Learning (ICML 2014","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2017a] Xinwang Liu, Miaomiao Li, Lei Wang, Yong Dou, Jianping Yin, and En Zhu. Multiple ker- nel k-means with incomplete kernels. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 2259-2265, 2017.","title":"Multiple kernel k-means with incomplete kernels","authors":"Liu","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2017b] Xinwang Liu, Sihang Zhou, Yueqing Wang, Miaomiao Li, Yong Dou, En Zhu, and Jianping Yin:. Optimal neighborhood kernel clustering with mul- tiple kernels. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 2266- 2272, 2017.","title":"Optimal neighborhood kernel clustering with multiple kernels","authors":"Liu","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2017c] Yong Liu, Shizhong Liao, Hailun Lin, Yinliang Yue, and Weiping Wang. Generalization anal- ysis for ranking using integral operator. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 2273-2279, 2017.","title":"Generalization analysis for ranking using integral operator","authors":"Liu","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2017d] Yong Liu, Shizhong Liao, Hailun Lin, Yinliang Yue, and Weiping Wang. Infinite kernel learning: generalization bounds and algorithms. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 2280-2286, 2017.","title":"Infinite kernel learning: generalization bounds and algorithms","authors":"Liu","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Luxburg et al., 2004] Ulrike Von Luxburg, Olivier Bous- quet, and Bernhard Schölkopf. A compression approach to support vector model selection. Journal of Machine Learning Research, 5:293-323, 2004.","title":"A compression approach to support vector model selection","authors":"Ulrike Von Luxburg, Olivier Bousquet, Bernhard Schölkopf","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nguyen and Ho, 2007] Canh Hao Nguyen and Tu Bao Ho. Kernel matrix evaluation. In Proceedings of the 20th In- ternational Joint Conference on Artifficial Intelligence (I- JCAI 2007), pages 987-992, 2007.","title":"Kernel matrix evaluation","authors":"Canh Hao, Nguyen, Tu Bao Ho","year":"2007","venue":"Proceedings of the 20th International Joint Conference on Artifficial Intelligence (I-JCAI 2007)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nguyen and Ho, 2008] Canh Hao Nguyen and Tu Bao Ho. An efficient kernel matrix evaluation measure. Pattern Recognition, 41(11):3366-3372, 2008.","title":"An efficient kernel matrix evaluation measure","authors":"Canh Hao, Nguyen, Tu Bao Ho","year":"2008","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shi and Malik, 2000] Jianbo Shi and Jitendra Malik. Nor- malized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888- 905, 2000.","title":"Normalized cuts and image segmentation","authors":"Jianbo Shi, Jitendra Malik","year":"2000","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Steinwart and Christmann, 2008] Ingo Steinwart and An- dreas Christmann. Support vector machines. Springer Ver- lag, New York, 2008.","title":"Ingo Steinwart and Andreas Christmann. Support vector machines","authors":"Christmann Steinwart","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vapnik, 2000] Vladimir Vapnik. The nature of statistical learning theory. Springer Verlag, 2000.","title":"The nature of statistical learning theory","authors":"Vladimir Vapnik","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wahba et al., 1999] Grace Wahba, Yi Lin, and Hao Zhang. GACV for support vector machines. In Advances in Large Margin Classifiers. MIT Press, Cambridge,, 1999.","title":"GACV for support vector machines","authors":"Grace Wahba, Yi Lin, Hao Zhang","year":"1999","venue":"Advances in Large Margin Classifiers","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang, 2002] Tong Zhang. Covering number bounds of cer- tain regularized linear function classes. Journal of Ma- chine Learning Research, 2:527-550, 2002.","title":"Covering number bounds of certain regularized linear function classes","authors":"Tong Zhang","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}