{"type":"ScholarlyArticle","title":"Data-driven Random Fourier Features using Stein Effect","authors":[{"name":"Wei-Cheng Chang","slug":"wei-cheng-chang","given":"Wei-Cheng","family":"Chang","affiliation":"CMU"},{"name":"Chun-Liang Li","slug":"chun-liang-li","given":"Chun-Liang","family":"Li","affiliation":"CMU"},{"name":"Yiming Yang","slug":"yiming-yang","given":"Yiming","family":"Yang","affiliation":""},{"name":"Barnabás Póczos","slug":"barnabas-poczos","given":"Barnabás","family":"Póczos","affiliation":"CMU"}],"abstract":"Large-scale kernel approximation is an important problem in machine learning research. Approaches using random Fourier features have become increasingly popular \\cite{Rahimi_NIPS_07}, where kernel approximation is treated as empirical mean estimation via Monte Carlo (MC) or Quasi-Monte Carlo (QMC) integration \\cite{Yang_ICML_14}. A limitation of the current approaches is that all the features receive an equal weight summing to 1. In this paper, we propose a novel shrinkage estimator from \"Stein effect\", which provides a data-driven weighting strategy for random features and enjoys theoretical justifications in terms of lowering the empirical risk. We further present an efficient randomized algorithm for large-scale applications of the proposed method. Our empirical results on six benchmark data sets demonstrate the advantageous performance of this approach over representative baselines in both kernel approximation and supervised learning tasks.","doi":"10.24963/ijcai.2017/207","url":"https://ijcai.org/proceedings/2017/207","pdf":"https://ijcai.org/proceedings/2017/0207.pdf","markdown":"https://ijcai.org/proceedings/2017/207.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/207","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":207,"pages":{"first":1497,"last":1503},"keywords":["Machine Learning: Machine Learning","Machine Learning: Feature Selection/Construction","Machine Learning: Kernel Methods"],"videos":[],"references":[{"position":1,"raw":"[Avron et al., 2013] Haim Avron, Vikas Sindhwani, and David Woodruff. Sketching structured matrices for faster nonlinear regression. In NIPS, 2013.","title":"Sketching structured matrices for faster nonlinear regression","authors":"Vikas Haim Avron, David Sindhwani, Woodruff","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avron et al., 2016a] Haim Avron, Kenneth L Clarkson, and David P Woodruff. Faster kernel ridge regression using sketching and preconditioning. arXiv, 2016.","title":"Faster kernel ridge regression using sketching and preconditioning","authors":"Avron","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Avron et al., 2016b] Haim Avron, Vikas Sindhwani, Jiyan Yang, and Michael W Mahoney. Quasi-monte carlo feature maps for shift-invariant kernels. JMLR, 2016.","title":"Quasi-monte carlo feature maps for shift-invariant kernels","authors":"Avron","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bach, 2015] Francis Bach. On the equivalence between quadrature rules and random features. arXiv, 2015.","title":"On the equivalence between quadrature rules and random features","authors":"Francis Bach","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caflisch, 1998] Russel E Caflisch. Monte carlo and quasi- monte carlo methods. Acta numerica, 1998.","title":"Monte carlo and quasimonte carlo methods","authors":"Russel E Caflisch","year":"1998","venue":"Acta numerica","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2011.","title":"Libsvm: a library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology (TIST)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chitta et al., 2012] Radha Chitta, Rong Jin, and Anil K Jain. Efficient kernel clustering using random fourier features. In ICDM, 2012.","title":"Efficient kernel clustering using random fourier features","authors":"Radha Chitta, Rong Jin, Anil K Jain","year":"2012","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2014] Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina F Balcan, and Le Song. Scalable kernel methods via doubly stochastic gradients. In NIPS, 2014.","title":"Scalable kernel methods via doubly stochastic gradients","authors":"Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina F Balcan, Le Song","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dick et al., 2013] Josef Dick, Frances Y Kuo, and Ian H Sloan. High-dimensional integration: the quasi-monte carlo way. Acta Numerica, 2013.","title":"High-dimensional integration: the quasi-monte carlo way","authors":"Josef Dick, Frances Y Kuo, Ian H Sloan","year":"2013","venue":"Acta Numerica","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Drineas and Mahoney, 2005] Petros Drineas and Michael W Mahoney. On the nyström method for approximating a gram matrix for improved kernel-based learning. JMLR, 2005.","title":"On the nyström method for approximating a gram matrix for improved kernel-based learning","authors":"Petros Drineas, Michael W Mahoney","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ghahramani and Rasmussen, 2002] Zoubin Ghahramani and Carl E Rasmussen. Bayesian monte carlo. In NIPS, 2002.","title":"Bayesian monte carlo","authors":"Zoubin Ghahramani, Carl E Rasmussen","year":"2002","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2014] Po-Sen Huang, Haim Avron, Tara N Sainath, Vikas Sindhwani, and Bhuvana Ramabhadran. Kernel methods match deep neural networks on timit. In ICASSP, 2014.","title":"Kernel methods match deep neural networks on timit","authors":"Huang","year":"2014","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huszár and Duvenaud, 2012] Ferenc Huszár and David Du- venaud. Optimally-weighted herding is bayesian quadra- ture. In UAI, 2012.","title":"Ferenc Huszár and David Duvenaud. Optimally-weighted herding is bayesian quadrature","authors":"Duvenaud Huszár","year":"2012","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kivinen et al., 2004] Jyrki Kivinen, Alexander J Smola, and Robert C Williamson. Online learning with kernels. IEEE transactions on signal processing, 52(8), 2004.","title":"Online learning with kernels","authors":"Jyrki Kivinen, Alexander J Smola, Robert C Williamson","year":"2004","venue":"IEEE transactions on signal processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le et al., 2013] Quoc Le, Tamás Sarlós, and Alexander Smola. Fastfood-computing hilbert space expansions in loglinear time. In ICML, 2013.","title":"Fastfood-computing hilbert space expansions in loglinear time","authors":"Quoc Le, Tamás Sarlós, Alexander Smola","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li and Póczos, 2016] Chun-Liang Li and Barnabás Póczos. Utilize old coordinates: Faster doubly stochastic gradients for kernel methods. In UAI, 2016.","title":"Utilize old coordinates: Faster doubly stochastic gradients for kernel methods","authors":"Chun-Liang Li, Barnabás Póczos","year":"2016","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Muandet et al., 2014] K. Muandet, K. Fukumizu, B. Sripe- rumbudur, A. Gretton, and B. Schölkopf. Kernel mean estimation and stein effect. In ICML, 2014. [Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NIPS, 2007.","title":"Kernel mean estimation and stein effect","authors":"K Muandet, K Fukumizu, B Sriperumbudur, A Gretton, B Schölkopf","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rahimi and Recht, 2009] Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing mini- mization with randomization in learning. In NIPS, 2009.","title":"Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rasmussen, 2006] Carl Edward Rasmussen. Gaussian pro- cesses for machine learning. 2006.","title":"Gaussian processes for machine learning","authors":"Carl Edward, Rasmussen","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rudi et al., 2016] Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Generalization properties of learning with random features. arXiv, 2016.","title":"Generalization properties of learning with random features","authors":"Alessandro Rudi, Raffaello Camoriano, Lorenzo Rosasco","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rudin, 2011] Walter Rudin. Fourier analysis on groups. John Wiley & Sons, 2011.","title":"Fourier analysis on groups","authors":"Walter Rudin","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sinha and Duchi, 2016] Aman Sinha and John C Duchi. Learning kernels with random features. In NIPS, 2016.","title":"Learning kernels with random features","authors":"Aman Sinha, John C Duchi","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Steinwart and Christmann, 2008] Ingo Steinwart and An- dreas Christmann. Support vector machines. Springer Science & Business Media, 2008.","title":"Ingo Steinwart and Andreas Christmann. Support vector machines","authors":"Christmann Steinwart","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang and Zhang, 2013] Shusen Wang and Zhihua Zhang. Improving cur matrix decomposition and the nyström ap- proximation via adaptive sampling. JMLR, 2013.","title":"Improving cur matrix decomposition and the nyström approximation via adaptive sampling","authors":"Wang, Zhang, Shusen Wang, Zhihua Zhang","year":"2013","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2017] Shusen Wang, Alex Gittens, and Michael W Mahoney. Sketched ridge regression: Opti- mization perspective, statistical perspective, and model averaging. arXiv, 2017.","title":"Sketched ridge regression: Optimization perspective, statistical perspective, and model averaging","authors":"Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Williams and Seeger, 2001] Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In NIPS, 2001.","title":"Using the nyström method to speed up kernel machines","authors":"Christopher Williams, Matthias Seeger","year":"2001","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Woodruff and others, 2014] David P Woodruff et al. Sketch- ing as a tool for numerical linear algebra. Foundations and Trends R in Theoretical Computer Science, 2014.","title":"Sketching as a tool for numerical linear algebra","authors":"P David, Woodruff","year":"2014","venue":"Foundations and Trends R in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2012] Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random fourier features: A theoretical and empirical comparison. In NIPS, 2012.","title":"Nyström method vs random fourier features: A theoretical and empirical comparison","authors":"Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, Zhi-Hua Zhou","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2014] Jiyan Yang, Vikas Sindhwani, Haim Avron, and Michael W Mahoney. Quasi-monte carlo fea- ture maps for shift-invariant kernels. In ICML, 2014.","title":"Quasi-monte carlo feature maps for shift-invariant kernels","authors":"Jiyan Yang, Vikas Sindhwani, Haim Avron, Michael W Mahoney","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}