{"type":"ScholarlyArticle","title":"Deep Spectral Kernel Learning","authors":[{"name":"Hui Xue","slug":"hui-xue","given":"Hui","family":"Xue","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; MOE Key Laboratory of Computer Network and Information Integration ( Southeast University) , China"},{"name":"Zheng-Fan Wu","slug":"zheng-fan-wu","given":"Zheng-Fan","family":"Wu","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; MOE Key Laboratory of Computer Network and Information Integration ( Southeast University) , China"},{"name":"Wei-Xiang Sun","slug":"wei-xiang-sun","given":"Wei-Xiang","family":"Sun","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; MOE Key Laboratory of Computer Network and Information Integration ( Southeast University) , China"}],"abstract":"Recently, spectral kernels have attracted wide attention in complex dynamic environments. These advanced kernels mainly focus on breaking through the crucial limitation on locality, that is, the stationarity and the monotonicity. But actually, owing to the inefficiency of shallow models in computational elements, they are more likely unable to accurately reveal dynamic and potential variations. In this paper, we propose a novel deep spectral kernel network (DSKN) to naturally integrate non-stationary and non-monotonic spectral kernels into elegant deep architectures in an interpretable way, which can be further generalized to cover most kernels. Concretely, we firstly deal with the general form of spectral kernels by the inverse Fourier transform. Secondly, DSKN is constructed by embedding the preeminent spectral kernels into each layer to boost the efficiency in computational elements, which can effectively reveal the dynamic input-dependent characteristics and potential long-range correlations by compactly representing complex advanced concepts. Thirdly, detailed analyses of DSKN are presented. Owing to its universality, we propose a unified spectral transform technique to flexibly extend and reasonably initialize domain-related DSKN. Furthermore, the representer theorem of DSKN is given. Systematical experiments demonstrate the superiority of DSKN compared to state-of-the-art relevant algorithms on varieties of standard real-world tasks.","doi":"10.24963/ijcai.2019/558","url":"https://ijcai.org/proceedings/2019/558","pdf":"https://ijcai.org/proceedings/2019/0558.pdf","markdown":"https://ijcai.org/proceedings/2019/558.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/558","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":558,"pages":{"first":4019,"last":4025},"keywords":["Machine Learning: Deep Learning","Machine Learning: Kernel Methods","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2006] Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of highly variable functions for local kernel machines. In Advances in neural informa- tion processing systems, pages 107-114, 2006.","title":"The curse of highly variable functions for local kernel machines","authors":"Yoshua Bengio, Olivier Delalleau, Nicolas Le Roux","year":"2006","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2007] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise train- ing of deep networks. In Advances in neural information processing systems, pages 153-160, 2007.","title":"Greedy layer-wise training of deep networks","authors":"Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blake and Merz, 1998] Catherine Blake and Christopher J Merz. Uci repository of machine learning databases. On- line at http://archive.ics.uci.edu/ml/, 1998.","title":"Uci repository of machine learning databases","authors":"Merz Blake, Catherine Blake, Christopher J Merz","year":"1998","venue":null,"doi":null,"url":"http://archive.ics.uci.edu/ml/,1998","internal_url":null},{"position":4,"raw":"[Bohn et al., 2017] Bastian Bohn, Michael Griebel, and Christian Rieger. A representer theorem for deep kernel learning. arXiv preprint arXiv:1709.10441, 2017.","title":"A representer theorem for deep kernel learning","authors":"Bastian Bohn, Michael Griebel, Christian Rieger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho and Saul, 2009] Youngmin Cho and Lawrence K Saul. Kernel methods for deep learning. In Advances in neural information processing systems, pages 342-350, 2009.","title":"Kernel methods for deep learning","authors":"Saul ; ] Youngmin Cho, Lawrence K Cho, Saul","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cutajar et al., 2017] Kurt Cutajar, Edwin V Bonilla, Pietro Michiardi, and Maurizio Filippone. Random feature ex- pansions for deep gaussian processes. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 884-893. JMLR. org, 2017.","title":"Random feature expansions for deep gaussian processes","authors":"Kurt Cutajar, Edwin V Bonilla, Pietro Michiardi, Maurizio Filippone","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Delalleau and Bengio, 2011] Olivier Delalleau and Yoshua Bengio. Shallow vs. deep sum-product networks. In Ad- vances in Neural Information Processing Systems, pages 666-674, 2011.","title":"Shallow vs. deep sum-product networks","authors":"Bengio Delalleau, Olivier Delalleau, Yoshua Bengio","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gibbs, 1998] Mark N Gibbs. Bayesian Gaussian Processes for Regression and Classification. PhD thesis, University of Cambridge, 1998.","title":"Bayesian Gaussian Processes for Regression and Classification","authors":"N Mark, Gibbs","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gikhman and Skorohod, 1974] Iosif Ilyich Gikhman and Anatolli Vladimirovich Skorohod. The Theory of Stochas- tic Processes, volume 1. Springer Verlag, Berlin, 1974.","title":"The Theory of Stochastic Processes","authors":"Iosif Ilyich, Gikhman, Anatolli Vladimirovich, Skorohod","year":"1974","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heinonen et al., 2016] Markus Heinonen, Henrik Manner- ström, Juho Rousu, Samuel Kaski, and Harri Lähdesmäki. Non-stationary gaussian process regression with hamilto- nian monte carlo. In Artificial Intelligence and Statistics, pages 732-740, 2016.","title":"Non-stationary gaussian process regression with hamiltonian monte carlo","authors":"Markus Heinonen, Henrik Mannerström, Juho Rousu, Samuel Kaski, Harri Lähdesmäki","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lázaro-Gredilla et al., 2010] Miguel Lázaro-Gredilla, Joaquin Quiñonero-Candela, Carl Edward Rasmussen, and Aníbal R Figueiras-Vidal. Sparse spectrum gaus- sian process regression. Journal of Machine Learning Research, 11:1865-1881, 2010.","title":"Sparse spectrum gaussian process regression","authors":"Miguel Lázaro-Gredilla, Joaquin Quiñonero-Candela, Carl Edward Rasmussen, Aníbal R Figueiras- Vidal","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rahimi and Recht, 2008] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Ad- vances in Neural Information Processing Systems, pages 1177-1184, 2008.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rasmussen and Williams, 2006] Carl Edward Rasmussen and Christopher K I Williams. Gaussian Process for Ma- chine Learning. MIT Press, 2006.","title":"Gaussian Process for Machine Learning","authors":"Carl Edward, Rasmussen, Christopher K I Williams","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rebai et al., 2016] Ilyes Rebai, Yassine BenAyed, and Walid Mahdi. Deep multilayer multiple kernel learning. Neural Computing and Applications, 27(8):2305-2314, 2016.","title":"Deep multilayer multiple kernel learning","authors":"Rebai","year":"2016","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Remes et al., 2017] Sami Remes, Markus Heinonen, and Samuel Kaski. Non-stationary spectral kernels. In Ad- vances in Neural Information Processing Systems, pages 4642-4651, 2017.","title":"Non-stationary spectral kernels","authors":"Sami Remes, Markus Heinonen, Samuel Kaski","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sampson and Guttorp, 1992] Paul D. Sampson and Peter Guttorp. Nonparametric estimation of nonstationary spa- tial covariance structure. Publications of the American Statistical Association, 87(417):108-119, 1992.","title":"Nonparametric estimation of nonstationary spatial covariance structure","authors":"Paul D Sampson, Peter Guttorp","year":"1992","venue":"Publications of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Stein, 1999] Michael L Stein. Interpolation of Spatial Data. Springer New York, 1999.","title":"Interpolation of Spatial Data","authors":"Michael L Stein","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2018] Shengyang Sun, Guodong Zhang, Chaoqi Wang, Wenyuan Zeng, Jiaman Li, and Roger Grosse. Dif- ferentiable compositional kernel learning for gaussian pro- cesses. arXiv preprint arXiv:1806.04326, 2018.","title":"Differentiable compositional kernel learning for gaussian processes","authors":"Shengyang Sun, Guodong Zhang, Chaoqi Wang, Wenyuan Zeng, Jiaman Li, Roger Grosse","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ton et al., 2018] Jean Francois Ton, Seth Flaxman, Dino Sejdinovic, and Samir Bhatt. Spatial mapping with gaus- sian processes and nonstationary fourier features. Spatial Statistics, 2018.","title":"Spatial mapping with gaussian processes and nonstationary fourier features","authors":"Jean Francois Ton, Seth Flaxman, Dino Sejdinovic, Samir Bhatt","year":"2018","venue":"Spatial Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wilson and Adams, 2013] Andrew Gordon Wilson and Ryan Prescott Adams. Gaussian process kernels for pat- tern discovery and extrapolation. In International Confer- ence on Machine Learning, pages 1067-1075, 2013.","title":"Gaussian process kernels for pattern discovery and extrapolation","authors":"Andrew Gordon, Wilson, Ryan Prescott, Adams","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wilson and Nickisch, 2015] Andrew Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International Conference on Machine Learning, pages 1775-1784, 2015.","title":"Kernel interpolation for scalable structured gaussian processes (kiss-gp)","authors":"Wilson, Nickisch, Andrew Wilson, Hannes Nickisch","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wilson et al., 2016a] Andrew G Wilson, Zhiting Hu, Rus- lan R Salakhutdinov, and Eric P Xing. Stochastic varia- tional deep kernel learning. In Advances in Neural Infor- mation Processing Systems, pages 2586-2594, 2016.","title":"Stochastic variational deep kernel learning","authors":"Wilson","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wilson et al., 2016b] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learn- ing. In Artificial Intelligence and Statistics, pages 370- 378, 2016.","title":"Deep kernel learning","authors":"Wilson","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2017] Hai-Ming Xu, Hui Xue, Xiao-Hong Chen, and Yun-Yun Wang. Solving indefinite kernel support vec- tor machine with difference of convex functions program- ming. In Thirty-First AAAI Conference on Artificial Intel- ligence, 2017.","title":"Solving indefinite kernel support vector machine with difference of convex functions programming","authors":"Hai-Ming Xu, Hui Xue, Xiao-Hong Chen, Yun-Yun Wang","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yaglom, 1987] Akira Moiseevich Yaglom. Correlation Theory of Stationary and Related Random Functions, vol- ume 1. Springer Series in Statistics, 1987.","title":"Akira Moiseevich Yaglom. Correlation Theory of Stationary and Related Random Functions","authors":"Yaglom","year":"1987","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Shuai Zhang, Jianxin Li, Pengtao Xie, Yingchun Zhang, Minglai Shao, Haoyi Zhou, and Mengyi Yan. Stacked kernel network. arXiv preprint arXiv:1711.09219, 2017.","title":"Stacked kernel network","authors":"Shuai Zhang, Jianxin Li, Pengtao Xie, Yingchun Zhang, Minglai Shao, Haoyi Zhou, Mengyi Yan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhuang et al., 2011] Jinfeng Zhuang, Ivor W Tsang, and Steven CH Hoi. Two-layer multiple kernel learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 909-917, 2011.","title":"Two-layer multiple kernel learning","authors":"Zhuang","year":"2011","venue":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}