{"type":"ScholarlyArticle","title":"Fast Cross-Validation","authors":[{"name":"Yong Liu","slug":"yong-liu","given":"Yong","family":"Liu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Hailun Lin","slug":"hailun-lin","given":"Hailun","family":"Lin","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Lizhong Ding","slug":"lizhong-ding","given":"Lizhong","family":"Ding","affiliation":"King Abdullah University of Science and Technology (KAUST)"},{"name":"Weiping Wang","slug":"weiping-wang","given":"Weiping","family":"Wang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Shizhong Liao","slug":"shizhong-liao","given":"Shizhong","family":"Liao","affiliation":"Tianjin University"}],"abstract":"Cross-validation (CV) is the most widely adopted approach for selecting the optimal model. However, the computation of CV  has high complexity due to multiple times of learner training, making it disabled for large scale model selection. In this paper, we present an approximate approach to CV based on the theoretical notion of Bouligand influence function (BIF) and the Nystr\\\"{o}m method for kernel methods. We first establish the relationship between the theoretical notion of BIF and CV, and propose a method to approximate the CV via the Taylor expansion of BIF. Then, we provide a novel computing method to calculate the BIF for general distribution, and evaluate BIF for sample distribution. Finally, we use the Nystr\\\"{o}m method to accelerate the computation of the BIF matrix for giving the finally approximate CV criterion. The proposed approximate CV requires training only once and is suitable for a wide variety of kernel methods. Experimental results on lots of datasets how that our approximate CV has no statistical discrepancy with the original CV, but can significantly improve the efficiency.","doi":"10.24963/ijcai.2018/346","url":"https://ijcai.org/proceedings/2018/346","pdf":"https://ijcai.org/proceedings/2018/0346.pdf","markdown":"https://ijcai.org/proceedings/2018/346.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/346","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":346,"pages":{"first":2497,"last":2503},"keywords":["Machine Learning: Machine Learning","Machine Learning Applications: Big data ; Scalability","Machine Learning: Data Mining","Machine Learning: Kernel Methods"],"videos":[],"references":[{"position":1,"raw":"[Cawley and Talbot, 2004] Gavin Cawley and Nicola Talbot. Fast leave-one-out cross-validation of sparse least-squares support vector machines. Neural Networks, 17(10):1467- 1475, 2004.","title":"Fast leave-one-out cross-validation of sparse least-squares support vector machines","authors":"Gavin Cawley, Nicola Talbot","year":"2004","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cawley and Talbot, 2007] Gavin Cawley and Nicola Tal- bot. Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters. Journal of Machine Learning Research, 8:841-861, 2007.","title":"Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters","authors":"Gavin Cawley, Nicola Talbot","year":"2007","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cawley and Talbot, 2008] Gavin Cawley and Nicola Talbot. Efficient approximate leave-one-out cross-validation for k- ernel logistic regression. Machine Learning, 71(2-3):243- 264, 2008.","title":"Efficient approximate leave-one-out cross-validation for kernel logistic regression","authors":"Gavin Cawley, Nicola Talbot","year":"2008","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cawley, 2006] Gavin Cawley. Leave-one-out cross- validation based model selection criteria for weighted LS- SVMs. In Proceedings of the International Joint Con- ference on Neural Networks (IJCNN 2006), pages 1661- 1668, 2006.","title":"Leave-one-out crossvalidation based model selection criteria for weighted LS-SVMs","authors":"Gavin Cawley","year":"2006","venue":"Proceedings of the International Joint Conference on Neural Networks (IJCNN 2006)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chapelle et al., 2002] Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, and Sayan Mukherjee. Choosing mul- tiple parameters for support vector machines. Machine Learning, 46(1-3):131-159, 2002.","title":"Choosing multiple parameters for support vector machines","authors":"Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, Sayan Mukherjee","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Christmann and Messem, 2008] Andreas Christmann and Arnout Van Messem. Bouligand derivatives and robust- ness of support vector machines for regression. Journal of Machine Learning Research, 9:915-936, 2008.","title":"Bouligand derivatives and robustness of support vector machines for regression","authors":"Messem Christmann, Andreas Christmann, Arnout Van Messem","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Debruyne et al., 2008] Michiel Debruyne, Mia Hubert, and Johan Suykens. Model selection in kernel based regression using the influence function. Journal of Machine Learning Research, 9:2377-2400, 2008.","title":"Model selection in kernel based regression using the influence function","authors":"Michiel Debruyne, Mia Hubert, Johan Suykens","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding and Liao, 2011] Lizhong Ding and Shizhong Liao. Approximate model selection for large scale LSSVM. Journal of Machine Learning Research -Proceedings Track, 20:165-180, 2011.","title":"Approximate model selection for large scale LSSVM","authors":"Liao ; ] Lizhong Ding, Shizhong Ding, Liao","year":"2011","venue":"Journal of Machine Learning Research -Proceedings Track","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ding and Liao, 2012] Lizhong Ding and Shizhong Liao. Nyström approximate model selection for LSSVM. In Proceedings of the 16th Pacific-Asia Conference (PAKDD 2012), pages 282-293, 2012.","title":"Nyström approximate model selection for LSSVM","authors":"Liao ; ] Lizhong Ding, Shizhong Ding, Liao","year":"2012","venue":"Proceedings of the 16th Pacific-Asia Conference (PAKDD 2012)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ding and Liao, 2014a] Lizhong Ding and Shizhong Liao. Approximate consistency: Towards foundations of ap- proximate kernel selection. In Proceedings of the Euro- pean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Database (ECML P- KDD), pages 354-369. Springer, Berlin, 2014.","title":"Approximate consistency: Towards foundations of approximate kernel selection","authors":"Liao Ding, Lizhong Ding, Shizhong Liao","year":"2014","venue":"Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Database (ECML P-KDD)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ding and Liao, 2014b] Lizhong Ding and Shizhong Liao. Model selection with the covering number of the ball of RKHS. In Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (CIKM 2014), pages 1159-1168, 2014.","title":"Model selection with the covering number of the ball of RKHS","authors":"Liao Ding, Lizhong Ding, Shizhong Liao","year":"2014","venue":"Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (CIKM 2014)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ding and Liao, 2017] Lizhong Ding and Shizhong Liao. An approximate approach to automatic kernel selection. IEEE Transactions on Cybernetics, 47(3):554-565, 2017.","title":"An approximate approach to automatic kernel selection","authors":"Liao ; ] Lizhong Ding, Shizhong Ding, Liao","year":"2017","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ding et al., 2018] Lizhong Ding, Shizhong Liao, Yong Liu, Peng Yang, and Xin Gao. Randomized kernel selection with spectra of multilevel circulant matrices. In Proceed- ings of the 32nd AAAI Conference on Artificial Intelligence (AAAI 2018), 2018.","title":"Randomized kernel selection with spectra of multilevel circulant matrices","authors":"Lizhong Ding, Shizhong Liao, Yong Liu, Peng Yang, Xin Gao","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI 2018)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hampel et al., 1986] Frank Hampel, Elvezio Ronchetti, Pe- ter Rousseeuw, and Werner Stahel. Robust statistics: the approach based on influence functions. Wiley, New York, 1986.","title":"Robust statistics: the approach based on influence functions","authors":"Hampel","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Koh and Liang, 2017] PangWei Koh and Percy Liang. Un- derstanding black-box predictions via influence functions. In Proceedings of the 34 th International Conference on Machine Learning (ICML 2017), pages 1885-1894, 2017.","title":"Understanding black-box predictions via influence functions","authors":"Liang Koh, Pangwei Koh, Percy Liang","year":"2017","venue":"Proceedings of the 34 th International Conference on Machine Learning (ICML 2017)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Jian Li, Yong Liu, Hailun Lin, Yinliang Yue, and Weiping Wang. Efficient kernel selection via spectral analysis. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI 2017), pages 2124-2130, 2017.","title":"Efficient kernel selection via spectral analysis","authors":"Jian Li, Yong Liu, Hailun Lin, Yinliang Yue, Weiping Wang","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI 2017)","doi":null,"url":null,"internal_url":"/proceedings/2017/295"},{"position":17,"raw":"[Liu and Liao, 2014] Yong Liu and Shizhong Liao. Prevent- ing over-fitting of cross-validation with kernel stability. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Dis- covery in Databases (ECML PKDD 2014), pages 290- 305, 2014.","title":"Preventing over-fitting of cross-validation with kernel stability","authors":"Yong Liu, Shizhong Liao","year":"2014","venue":"Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2014)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu and Liao, 2015] Yong Liu and Shizhong Liao. Eigen- values ratio for kernel selection of kernel methods. In Pro- ceedings of the Twenty-Ninth AAAI Conference on Artifi- cial Intelligence (AAAI 2015), pages 2814-2820, 2015.","title":"Eigenvalues ratio for kernel selection of kernel methods","authors":"Yong Liu, Shizhong Liao","year":"2015","venue":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI 2015)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2013] Yong Liu, Shali Jiang, and Shizhong Liao. Eigenvalues perturbation of integral operator for kernel s- election. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013), pages 2189-2198, 2013.","title":"Eigenvalues perturbation of integral operator for kernel selection","authors":"Yong Liu, Shali Jiang, Shizhong Liao","year":"2013","venue":"Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2014] Yong Liu, Shali Jiang, and Shizhong Liao. Efficient approximation of cross-validation for kernel methods using Bouligand influence function. In Proceed- ings of The 31st International Conference on Machine Learning (ICML 2014 (1)), pages 324-332, 2014.","title":"Efficient approximation of cross-validation for kernel methods using Bouligand influence function","authors":"Yong Liu, Shali Jiang, Shizhong Liao","year":"2014","venue":"Proceedings of The 31st International Conference on Machine Learning (ICML 2014","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2017] Yong Liu, Shizhong Liao, Hailun Lin, Yin- liang Yue, and Weiping Wang. Infinite kernel learning: generalization bounds and algorithms. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI 2017), 2017.","title":"Infinite kernel learning: generalization bounds and algorithms","authors":"Yong Liu, Shizhong Liao, Hailun Lin, Yinliang Yue, Weiping Wang","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI 2017)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Robinson, 1991] Stephen Robinson. An implicit-function theorem for a class of nonsmooth functions. Mathematics of Operations Research, 16:292-309, 1991.","title":"An implicit-function theorem for a class of nonsmooth functions","authors":"Stephen Robinson","year":"1991","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Steinwart and Christmann, 2008] Ingo Steinwart and An- dreas Christmann. Support vector machines. Springer Ver- lag, 2008.","title":"Ingo Steinwart and Andreas Christmann. Support vector machines","authors":"Christmann Steinwart","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vapnik and Chapelle, 2000] Vladimir Vapnik and Olivier Chapelle. Bounds on error expectation for support vector machines. Neural Computation, 12(9):2013-2036, 2000.","title":"Bounds on error expectation for support vector machines","authors":"Chapelle ; Vapnik, Olivier Chapelle","year":"2000","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vito et al., 2004] Ernesto De Vito, Lorenzo Rosasco, An- drea Caponnetto, Michele Piana, and Alessandro Verri. Some properties of regularized kernel methods. The Jour- nal of Machine Learning Research, 5:1363-1390, 2004.","title":"Some properties of regularized kernel methods","authors":"Ernesto De, Vito, Lorenzo Rosasco, Andrea Caponnetto, Michele Piana, Alessandro Verri","year":"2004","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}