{"type":"ScholarlyArticle","title":"Hypothesis Sketching for Online Kernel Selection in Continuous Kernel Space","authors":[{"name":"Xiao Zhang","slug":"xiao-zhang","given":"Xiao","family":"Zhang","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin 300350 , China"},{"name":"Shizhong Liao","slug":"shizhong-liao","given":"Shizhong","family":"Liao","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin 300350 , China"}],"abstract":"Online kernel selection in continuous kernel space is more complex than that in discrete kernel set. But existing online kernel selection approaches for continuous kernel spaces have linear computational complexities at each round with respect to the current number of rounds and lack sublinear regret guarantees due to the continuously many candidate kernels. To address these issues, we propose a novel hypothesis sketching approach to online kernel selection in continuous kernel space, which has constant computational complexities at each round and enjoys a sublinear regret bound. The main idea of the proposed hypothesis sketching approach is to maintain the orthogonality of the basis functions and the prediction accuracy of the hypothesis sketches in a time-varying reproducing kernel Hilbert space. We first present an efficient dependency condition to maintain the basis functions of the hypothesis sketches under a computational budget. Then we update the weights and the optimal kernels by minimizing the instantaneous loss of the hypothesis sketches using the online gradient descent with a compensation strategy. We prove that the proposed hypothesis sketching approach enjoys a regret bound of order O(√T) for online kernel selection in continuous kernel space, which is optimal for convex loss functions, where T is the number of rounds, and reduces the computational complexities at each round from linear to constant with respect to the number of rounds. Experimental results demonstrate that the proposed hypothesis sketching approach significantly improves the efficiency of online kernel selection in continuous kernel space while retaining comparable predictive accuracies.","doi":"10.24963/ijcai.2020/346","url":"https://ijcai.org/proceedings/2020/346","pdf":"https://ijcai.org/proceedings/2020/0346.pdf","markdown":"https://ijcai.org/proceedings/2020/346.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/346","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":346,"pages":{"first":2498,"last":2504},"keywords":["Machine Learning: Kernel Methods","Machine Learning: Online Learning","Machine Learning: Learning Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24148/file_24148.m3u8","length_sec":268,"provider":"hls"}],"references":[{"position":1,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Auer","year":"2002","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cesa-Bianchi and Lugosi, 2006] Nicolò Cesa-Bianchi and Gabor Lugosi. Prediction, learning, and games. Cam- bridge University Press, 2006.","title":"Nicolò Cesa-Bianchi and Gabor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2016] Badong Chen, Junli Liang, Nanning Zheng, and José C Príncipe. Kernel least mean square with adaptive kernel size. Neurocomputing, 191:95-106, 2016.","title":"Kernel least mean square with adaptive kernel size","authors":"Badong Chen, Junli Liang, Nanning Zheng, José C Príncipe","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cortes et al., 2012] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. Journal of Machine Learn- ing Research, 13:795-828, 2012.","title":"Algorithms for learning kernels based on centered alignment","authors":"Corinna Cortes, Mehryar Mohri, Afshin Rostamizadeh","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Crammer et al., 2003] Koby Crammer, Jaz Kandola, and Yoram Singer. Online classification on a budget. In Advances in Neural Information Processing Systems 16, pages 225-232, 2003.","title":"Online classification on a budget","authors":"Koby Crammer, Jaz Kandola, Yoram Singer","year":"2003","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Diethe and Girolami, 2013] Tom Diethe and Mark Giro- lami. Online learning with (multiple) kernels: A review. Neural Computation, 25(3):567-625, 2013.","title":"Tom Diethe and Mark Girolami. Online learning with (multiple) kernels: A review","authors":"Girolami Diethe","year":"2013","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ding et al., 2018] Lizhong Ding, Shizhong Liao, Yong Liu, Peng Yang, and Xin Gao. Randomized kernel selection with spectra of multilevel circulant matrices. In Proceed- ings of the 32nd AAAI Conference on Artificial Intelli- gence, pages 2910-2917, 2018.","title":"Randomized kernel selection with spectra of multilevel circulant matrices","authors":"Lizhong Ding, Shizhong Liao, Yong Liu, Peng Yang, Xin Gao","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Foster et al., 2017] Dylan J Foster, Satyen Kale, Mehryar Mohri, and Karthik Sridharan. Parameter-free online learning via model selection. In Advances in Neural Infor- mation Processing Systems 30, pages 6022-6032, 2017.","title":"Parameter-free online learning via model selection","authors":"Dylan J Foster, Satyen Kale, Mehryar Mohri, Karthik Sridharan","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Freund and Schapire, 1999] Yoav Freund and Robert E Schapire. Large margin classification using the perceptron algorithm. Machine learning, 37(3):277-296, 1999.","title":"Large margin classification using the perceptron algorithm","authors":"Schapire Freund, Yoav Freund, Robert E Schapire","year":"1999","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hazan, 2016] Elad Hazan. Introduction to online convex optimization. Foundations and Trends R in Optimization, 2(3-4):157-325, 2016.","title":"Introduction to online convex optimization","authors":"Elad Hazan","year":"2016","venue":"Foundations and Trends R in Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2015] Junjie Hu, Haiqin Yang, Irwin King, Michael R Lyu, and Anthony Man-Cho So. Kernelized online imbalanced learning with fixed budgets. In Pro- ceedings of the 29th AAAI Conference on Artificial Intelli- gence, pages 2666-2672, 2015.","title":"Kernelized online imbalanced learning with fixed budgets","authors":"Junjie Hu, Haiqin Yang, Irwin King, Anthony Man-Cho Michael R Lyu, So","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kandola et al., 2002] J. Kandola, J. Shawe-Taylor, and N. Cristianini. On the extensions of kernel alignment. Technical Report Technical report 120, University of Lon- don, 2002.","title":"On the extensions of kernel alignment","authors":"J Kandola, J Shawe-Taylor, N Cristianini","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kivinen et al., 2001] Jyrki Kivinen, Alex J Smola, and Robert C Williamson. Online learning with kernels. In Advances in Neural Information Processing Systems 14, pages 785-792, 2001.","title":"Online learning with kernels","authors":"Jyrki Kivinen, Alex J Smola, Robert C Williamson","year":"2001","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2014] Yong Liu, Shali Jiang, and Shizhong Liao. Efficient approximation of cross-validation for kernel methods using bouligand influence function. In Proceed- ings of the 31st International Conference on Machine Learning, pages 324-332, 2014.","title":"Efficient approximation of cross-validation for kernel methods using bouligand influence function","authors":"Yong Liu, Shali Jiang, Shizhong Liao","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018] Yong Liu, Hailun Lin, Lizhong Ding, Weiping Wang, and Shizhong Liao. Fast cross-validation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2497-2503, 2018.","title":"Fast cross-validation","authors":"Yong Liu, Hailun Lin, Lizhong Ding, Weiping Wang, Shizhong Liao","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Minh, 2010] Ha Quang Minh. Some properties of Gaussian reproducing kernel Hilbert spaces and their implications for function approximation and learning theory. Construc- tive Approximation, 32(2):307-338, 2010.","title":"Some properties of Gaussian reproducing kernel Hilbert spaces and their implications for function approximation and learning theory","authors":"Ha Quang, Minh","year":"2010","venue":"Constructive Approximation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen et al., 2017] Tu Dinh Nguyen, Trung Le, Hung Bui, and Dinh Q. Phung. Large-scale online kernel learning with random feature reparameterization. In Proceedings of the 26th International Joint Conference on Artificial In- telligence, pages 2543-2549, 2017.","title":"Large-scale online kernel learning with random feature reparameterization","authors":"Trung Tu Dinh Nguyen, Hung Le, Dinh Q Bui, Phung","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/354"},{"position":18,"raw":"[Orabona et al., 2008] Francesco Orabona, Joseph Keshet, and Barbara Caputo. The projectron: A bounded kernel- based perceptron. In Proceedings of the 25th International Conference on Machine learning, pages 720-727, 2008.","title":"The projectron: A bounded kernelbased perceptron","authors":"Francesco Orabona, Joseph Keshet, Barbara Caputo","year":"2008","venue":"Proceedings of the 25th International Conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shalev-Shwartz, 2011] Shai Shalev-Shwartz. Online learn- ing and online convex optimization. Foundations and Trends R in Machine Learning, 4(2):107-194, 2011.","title":"Online learning and online convex optimization","authors":"Shai Shalev-Shwartz","year":"2011","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Abhishek Singh and José C. Príncipe. Information theoretic learning with adaptive ker- nels. IEEE Transactions on Signal Processing, 91(2):203- 213, 2011.","title":"Information theoretic learning with adaptive kernels","authors":"Abhishek Singh, José C Príncipe","year":"2011","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2012] Tianbao Yang, Mehrdad Mahdavi, Rong Jin, Jinfeng Yi, and Steven C.H. Hoi. Online kernel se- lection: Algorithms and evaluations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence, pages 22-26, 2012.","title":"Online kernel selection: Algorithms and evaluations","authors":"Tianbao Yang, Mehrdad Mahdavi, Rong Jin, Jinfeng Yi, Steven C H Hoi","year":"2012","venue":"Proceedings of the 26th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yukawa, 2015] Masahiro Yukawa. Adaptive learning in Cartesian product of reproducing kernel Hilbert spaces. IEEE Transactions on Signal Processing, 63(22):6037- 6048, 2015.","title":"Adaptive learning in Cartesian product of reproducing kernel Hilbert spaces","authors":"Masahiro Yukawa","year":"2015","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and Liao, 2018] Xiao Zhang and Shizhong Liao. On- line kernel selection via incremental sketched kernel align- ment. In Proceedings of the 27th International Joint Con- ference on Artificial Intelligence, pages 3118-3124, 2018.","title":"Online kernel selection via incremental sketched kernel alignment","authors":"Xiao Zhang, Shizhong Liao","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/433"},{"position":24,"raw":"[Zhang and Liao, 2019] Xiao Zhang and Shizhong Liao. In- cremental randomized sketching for online kernel learn- ing. In Proceedings of the 36th International Conference on Machine Learning, pages 7394-7403, 2019.","title":"Incremental randomized sketching for online kernel learning","authors":"Xiao Zhang, Shizhong Liao","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019] Xiao Zhang, Yun Liao, and Shizhong Liao. A survey on online kernel selection for online kernel learning. WIREs Data Mining and Knowledge Discovery, 9(2):e1295, 2019.","title":"A survey on online kernel selection for online kernel learning","authors":"Xiao Zhang, Yun Liao, Shizhong Liao","year":"2019","venue":"WIREs Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2012] Peilin Zhao, Jialei Wang, Pengcheng Wu, Rong Jin, and Steven C.H. Hoi. Fast bounded online gra- dient descent algorithms for scalable kernel-based online learning. In Proceedings of the 29th International Confer- ence on Machine Learning, pages 169-176, 2012.","title":"Fast bounded online gradient descent algorithms for scalable kernel-based online learning","authors":"Peilin Zhao, Jialei Wang, Pengcheng Wu, Rong Jin, Steven C H Hoi","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Ma- chine Learning, pages 928-936, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"Proceedings of the 20th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}