{"type":"ScholarlyArticle","title":"Large-scale Online Kernel Learning with Random Feature Reparameterization","authors":[{"name":"Tu Dinh Nguyen","slug":"tu-dinh-nguyen","given":"Tu Dinh","family":"Nguyen","affiliation":"‡Adobe Research , Adobe Systems, Inc. †Center for Pattern Recognition and Data Analytics , Deakin University , Australia"},{"name":"Trung Le","slug":"trung-le","given":"Trung","family":"Le","affiliation":"‡Adobe Research , Adobe Systems, Inc. †Center for Pattern Recognition and Data Analytics , Deakin University , Australia"},{"name":"Hung Bui","slug":"hung-bui","given":"Hung","family":"Bui","affiliation":"‡Adobe Research , Adobe Systems, Inc. †Center for Pattern Recognition and Data Analytics , Deakin University , Australia"},{"name":"Dinh Phung","slug":"dinh-phung","given":"Dinh","family":"Phung","affiliation":"‡Adobe Research , Adobe Systems, Inc. †Center for Pattern Recognition and Data Analytics , Deakin University , Australia"}],"abstract":"A typical online kernel learning method faces two fundamental issues: the complexity in dealing with a huge number of observed data points (a.k.a the curse of kernelization) and the difficulty in learning kernel parameters, which often assumed to be fixed. Random Fourier feature is a recent and effective approach to address the former by approximating the shift-invariant kernel function via Bocher's theorem, and allows the model to be maintained directly in the random feature space with a fixed dimension, hence the model size remains constant w.r.t. data size. We further introduce in this paper the reparameterized random feature (RRF), a random feature framework for large-scale online kernel learning to address both aforementioned challenges. Our initial intuition comes from the so-called \"reparameterization trick\" [Kingma et al., 2014] to lift the source of randomness of Fourier components to another space which can be independently sampled, so that stochastic gradient of the kernel parameters can be analytically derived. We develop a well-founded underlying theory for our method, including a general way to reparameterize the kernel, and a new tighter error bound on the approximation quality. This view further inspires a direct application of stochastic gradient descent for updating our model under an online learning setting. We then conducted extensive experiments on several large-scale datasets where we demonstrate that our work achieves state-of-the-art performance in both learning efficacy and efficiency.","doi":"10.24963/ijcai.2017/354","url":"https://ijcai.org/proceedings/2017/354","pdf":"https://ijcai.org/proceedings/2017/0354.pdf","markdown":"https://ijcai.org/proceedings/2017/354.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/354","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":354,"pages":{"first":2543,"last":2549},"keywords":["Machine Learning: Kernel Methods","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Bȃzȃvan et al., 2012] Eduard Gabriel Bȃzȃvan, Fuxin Li, and Cristian Sminchisescu. Fourier kernel learning. In European Conference on Computer Vision (ECCV), pages 459-473. Springer, 2012.","title":"Fourier kernel learning","authors":"Bȃzȃvan","year":"2012","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bochner, 1959] Salomon Bochner. Lectures on Fourier In- tegrals, volume 42. Princeton University Press, 1959.","title":"Salomon Bochner","authors":"Bochner","year":"1959","venue":"Lectures on Fourier Integrals","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cavallanti et al., 2007] Giovanni Cavallanti, Nicolo Cesa- Bianchi, and Claudio Gentile. Tracking the best hyper- plane with a simple budget perceptron. Machine Learning, 69(2-3):143-167, 2007.","title":"Tracking the best hyperplane with a simple budget perceptron","authors":"Giovanni Cavallanti, Nicolo Cesa-Bianchi, Claudio Gentile","year":"2007","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chapelle et al., 2002] Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, and Sayan Mukherjee. Choosing mul- tiple parameters for support vector machines. Machine learning, 46(1-3):131-159, 2002.","title":"Choosing multiple parameters for support vector machines","authors":"Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, Sayan Mukherjee","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Crammer and Singer, 2001] Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learn- ing Research (JMLR), 2:265-292, December 2001.","title":"On the algorithmic implementation of multiclass kernel-based vector machines","authors":"Koby Crammer, Yoram Singer","year":"2001","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Crammer et al., 2006] Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. Online passive-aggressive algorithms. Journal of Machine Learn- ing Research (JMLR), 7:551-585, 2006.","title":"Online passive-aggressive algorithms","authors":"Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, Yoram Singer","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dekel et al., 2005] Ofer Dekel, Shai Shalev-Shwartz, and Yoram Singer. The forgetron: A kernel-based perceptron on a fixed budget. In Advances in Neural Information Pro- cessing Systems (NIPS), pages 259-266, 2005.","title":"The forgetron: A kernel-based perceptron on a fixed budget","authors":"Ofer Dekel, Shai Shalev-Shwartz, Yoram Singer","year":"2005","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Freund and Schapire, 1999] Yoav Freund and Robert E Schapire. Large margin classification using the perceptron algorithm. Machine learning, 37(3):277-296, 1999.","title":"Large margin classification using the perceptron algorithm","authors":"Schapire Freund, Yoav Freund, Robert E Schapire","year":"1999","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hensman et al., 2013] James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. In Un- certainty in Artificial Intelligence (UAI), pages 1-9, 2013.","title":"Gaussian processes for big data","authors":"James Hensman, Nicolo Fusi, Neil D Lawrence","year":"2013","venue":"Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In The Inter. Conf. on Learning Representations (ICLR), 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"The Inter. Conf. on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kivinen et al., 2004] Jyrki Kivinen, Alexander J Smola, and Robert C Williamson. Online learning with kernels. IEEE Transactions on Signal Processing, 52(8):2165- 2176, Aug 2004.","title":"Online learning with kernels","authors":"Jyrki Kivinen, Alexander J Smola, Robert C Williamson","year":"2004","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Le et al., 2013] Quoc Le, Tamás Sarlós, and Alex Smola. Fastfood -approximating kernel expansions in loglinear time. In Proceedings of the international conference on machine learning (ICML), pages 1281-1289, 2013.","title":"Fastfood -approximating kernel expansions in loglinear time","authors":"Quoc Le, Tamás Sarlós, Alex Smola","year":"2013","venue":"Proceedings of the international conference on machine learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Le et al., 2016a] Trung Le, Phuong Duong, Mi Dinh, Tu D Nguyen, Vu Nguyen, and Dinh Phung. Budgeted semi- supervised support vector machine. In Uncertainty in Ar- tificial Intelligence (UAI), pages 377-386, 2016.","title":"Budgeted semisupervised support vector machine","authors":"Le","year":"2016","venue":"Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Le et al., 2016b] Trung Le, Tu D Nguyen, Vu Nguyen, and Dinh Phung. Dual space gradient descent for online learn- ing. In Advances in Neural Information Processing Sys- tems 29 (NIPS), pages 4583-4591. 2016.","title":"Dual space gradient descent for online learning","authors":"Le","year":"2016","venue":"Advances in Neural Information Processing Systems 29 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le et al., 2016c] Trung Le, Vu Nguyen, Tu D Nguyen, and Dinh Phung. Nonparametric budgeted stochastic gradient descent. In 19th International Conference on Artificial In- telligence and Statistics (AISTATS), pages 654-572, 2016.","title":"Nonparametric budgeted stochastic gradient descent","authors":"Le","year":"2016","venue":"19th International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2015] Jing Lu, Steven CH Hoi, Jialei Wang, Peilin Zhao, and Zhi-Yong Liu. Large scale online kernel learn- ing. Journal of Machine Learning Research (JMLR), 2015.","title":"Large scale online kernel learning","authors":"Jing Lu, C H Steven, Jialei Hoi, Peilin Wang, Zhi-Yong Zhao, Liu","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moeller et al., 2016] John Moeller, Vivek Srikumar, Sarathkrishna Swaminathan, Suresh Venkatasubramanian, and Dustin Webb. Continuous kernel learning. In Euro- pean Conference on Machine Learning (ECML), pages 657-673. Springer, 2016.","title":"Continuous kernel learning","authors":"John Moeller, Vivek Srikumar, Sarathkrishna Swaminathan, Suresh Venkatasubramanian, Dustin Webb","year":"2016","venue":"European Conference on Machine Learning (ECML)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nguyen et al., 2016] Khanh Nguyen, Trung Le, Vu Nguyen, Tu D Nguyen, and Dinh Phung. Mul- tiple kernel learning with data augmentation. In the 8th Asian Conference on Machine Learning (ACML), volume 63, pages 49-64, 16-18 Nov 2016.","title":"Multiple kernel learning with data augmentation","authors":"Khanh Nguyen, Trung Le, Vu Nguyen, Tu D Nguyen, Dinh Phung","year":"2016","venue":"the 8th Asian Conference on Machine Learning (ACML)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oliva et al., 2015] Junier Oliva, Avinava Dubey, Barn- abas Poczos, Jeff Schneider, and Eric P Xing. Bayesian nonparametric kernel-learning. arXiv preprint arXiv:1506.08776, 2015.","title":"Bayesian nonparametric kernel-learning","authors":"Oliva","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Orabona et al., 2009] Francesco Orabona, Joseph Keshet, and Barbara Caputo. Bounded kernel-based online learn- ing. Journal of Machine Learning Research (JMLR), 10:2643-2666, 2009.","title":"Bounded kernel-based online learning","authors":"Francesco Orabona, Joseph Keshet, Barbara Caputo","year":"2009","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Neu- ral Information Processing Systems, volume 3, 2007.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2007","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Vucetic, 2009] Zhuang Wang and Slobodan Vucetic. Twin vector machines for online learning on a budget. In SDM, pages 906-917. SIAM, 2009.","title":"Twin vector machines for online learning on a budget","authors":"Wang, Vucetic ;, Zhuang Wang, Slobodan Vucetic","year":"2009","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Vucetic, 2010] Zhuang Wang and Slobodan Vucetic. Online passive-aggressive algorithms on a budget. In AISTATS, volume 9, pages 908-915, 2010.","title":"Online passive-aggressive algorithms on a budget","authors":"Zhuang Wang, Slobodan Vucetic","year":"2010","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2012] Zhuang Wang, Koby Crammer, and Slo- bodan Vucetic. Breaking the curse of kernelization: Bud- geted stochastic gradient descent for large-scale svm train- ing. JMLR, 13(1):3103-3131, 2012.","title":"Breaking the curse of kernelization: Budgeted stochastic gradient descent for large-scale svm training","authors":"Zhuang Wang, Koby Crammer, Slobodan Vucetic","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wilson and Adams, 2013] Andrew Gordon Wilson and Ryan Prescott Adams. Gaussian process kernels for pat- tern discovery and extrapolation. In International Confer- ence in Machine Learning (ICML) 2013, volume 28, pages 1067-1075, 2013.","title":"Gaussian process kernels for pattern discovery and extrapolation","authors":"Andrew Gordon, Wilson, Ryan Prescott, Adams","year":"2013","venue":"International Conference in Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2015] Zichao Yang, Alexander J Smola, Le Song, and Andrew Gordon Wilson. A la carte-learning fast kernels. In 18th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 38, pages 1098-1106, San Diego, USA, 9-12 May 2015.","title":"A la carte-learning fast kernels","authors":"Yang","year":"2015","venue":"18th International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2012] Peilin Zhao, Jialei Wang, Pengcheng Wu, Rong Jin, and Steven CH Hoi. Fast bounded online gra- dient descent algorithms for scalable kernel-based online learning. arXiv preprint arXiv:1206.4633, 2012.","title":"Fast bounded online gradient descent algorithms for scalable kernel-based online learning","authors":"Peilin Zhao, Jialei Wang, Pengcheng Wu, Rong Jin, Steven Ch Hoi","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning (ICML). Carnegie Mellon University, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}