{"type":"ScholarlyArticle","title":"BaKer-Nets: Bayesian Random Kernel Mapping Networks","authors":[{"name":"Hui Xue","slug":"hui-xue","given":"Hui","family":"Xue","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; Key Laboratory of Computer Network and Information Integration (Southeast University) , Ministry of Education , Nanjing , 210096 , China"},{"name":"Zheng-Fan Wu","slug":"zheng-fan-wu","given":"Zheng-Fan","family":"Wu","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; Key Laboratory of Computer Network and Information Integration (Southeast University) , Ministry of Education , Nanjing , 210096 , China"}],"abstract":"Recently, deep spectral kernel networks (DSKNs) have attracted wide attention. They consist of periodic computational elements that can be activated across the whole feature spaces. In theory, DSKNs have the potential to reveal input-dependent and long-range characteristics, and thus are expected to perform more competitive than prevailing networks. But in practice, they are still unable to achieve the desired effects. The structural superiority of DSKNs comes at the cost of the difficult optimization. The periodicity of computational elements leads to many poor and dense local minima in loss landscapes. DSKNs are more likely stuck in these local minima, and perform worse than expected. Hence, in this paper, we propose the novel Bayesian random Kernel mapping Networks (BaKer-Nets) with preferable learning processes by escaping randomly from most local minima. Specifically, BaKer-Nets consist of two core components: 1) a prior-posterior bridge is derived to enable the uncertainty of computational elements reasonably; 2) a Bayesian learning paradigm is presented to optimize the prior-posterior bridge efficiently. With the well-tuned uncertainty, BaKer-Nets can not only explore more potential solutions to avoid local minima, but also exploit these ensemble solutions to strengthen their robustness. Systematical experiments demonstrate the significance of BaKer-Nets in improving learning processes on the premise of preserving the structural superiority.","doi":"10.24963/ijcai.2020/425","url":"https://ijcai.org/proceedings/2020/425","pdf":"https://ijcai.org/proceedings/2020/0425.pdf","markdown":"https://ijcai.org/proceedings/2020/425.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/425","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":425,"pages":{"first":3073,"last":3079},"keywords":["Machine Learning: Deep Learning","Machine Learning: Kernel Methods"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24317/file_24317.m3u8","length_sec":294,"provider":"hls"}],"references":[{"position":1,"raw":"[Aas et al., 2006] Kjersti Aas, Claudia Czado, Arnoldo Frigessi, and Henrik Bakken. Pair-copula constructions of multiple dependence. Insurance Mathematics & Eco- nomics, 44(2):182-198, 2006.","title":"Pair-copula constructions of multiple dependence","authors":"Kjersti Aas, Claudia Czado, Arnoldo Frigessi, Henrik Bakken","year":"2006","venue":"Insurance Mathematics & Economics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2006] Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of highly variable functions for local kernel machines. In Advances in neural informa- tion processing systems, pages 107-114, 2006.","title":"The curse of highly variable functions for local kernel machines","authors":"Yoshua Bengio, Olivier Delalleau, Nicolas Le Roux","year":"2006","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2007a] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise train- ing of deep networks. In Advances in neural information processing systems, pages 153-160, 2007.","title":"Greedy layer-wise training of deep networks","authors":"Bengio","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2007b] Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale ker- nel machines, 34(5):1-41, 2007.","title":"Scaling learning algorithms towards ai","authors":"Bengio","year":"2007","venue":"Large-scale kernel machines","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blake and Merz, 1998] Catherine Blake and Christopher J Merz. Uci repository of machine learning databases. On- line at http://archive.ics.uci.edu/ml/, 1998.","title":"Uci repository of machine learning databases","authors":"Merz Blake, Catherine Blake, Christopher J Merz","year":"1998","venue":null,"doi":null,"url":"http://archive.ics.uci.edu/ml/,1998","internal_url":null},{"position":6,"raw":"[Casella and Robert, 1996] George Casella and Christian P Robert. Rao-blackwellisation of sampling schemes. Biometrika, 83(1):81-94, 1996.","title":"Rao-blackwellisation of sampling schemes","authors":"George Casella, Christian P Robert","year":"1996","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011. Software available at http://www. csie.ntu.edu.tw/ ∼ cjlin/libsvm.","title":"LIBSVM: A library for support vector machines","authors":"Chang, Lin ; Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":"http://www.csie.ntu.edu.tw/∼cjlin/libsvm","internal_url":null},{"position":8,"raw":"[Charpentier and Segers, 2009] Arthur Charpentier and Jo- han Segers. Tails of multivariate archimedean copu- las. Journal of Multivariate Analysis, 100(7):1521-1537, 2009.","title":"Tails of multivariate archimedean copulas","authors":"Segers Charpentier","year":"2009","venue":"Journal of Multivariate Analysis","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Delalleau and Bengio, 2011] Olivier Delalleau and Yoshua Bengio. Shallow vs. deep sum-product networks. In Ad- vances in Neural Information Processing Systems, pages 666-674, 2011.","title":"Shallow vs. deep sum-product networks","authors":"Bengio Delalleau, Olivier Delalleau, Yoshua Bengio","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth in- ternational conference on artificial intelligence and statis- tics, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Proceedings of the thirteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li, 2000] David X Li. On default correlation: A copula function approach. Social Science Electronic Publishing, 9(4), 2000.","title":"On default correlation: A copula function approach","authors":"Li David","year":"2000","venue":"Social Science Electronic Publishing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mandt and Blei, 2014] Stephan Mandt and David Blei. Smoothed gradients for stochastic variational inference. In Advances in Neural Information Processing Systems, pages 2438-2446, 2014.","title":"Smoothed gradients for stochastic variational inference","authors":"Blei Mandt, Stephan Mandt, David Blei","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ranganath et al., 2014] Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Ar- tificial Intelligence and Statistics, pages 814-822, 2014.","title":"Black box variational inference","authors":"Rajesh Ranganath, Sean Gerrish, David Blei","year":"2014","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ross, 1994] Sheldon M Ross. A new simulation estimator of system reliability. International Journal of Stochastic Analysis, 7(3):331-336, 1994.","title":"A new simulation estimator of system reliability","authors":"M Sheldon, Ross","year":"1994","venue":"International Journal of Stochastic Analysis","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sklar, 1959] M Sklar. Fonctions de répartition À n dimen- sions et leurs marges. Publ.inst.statist.univ, Paris, 8:229- 231, 1959.","title":"Fonctions de répartition À n dimensions et leurs marges","authors":"Sklar","year":"1959","venue":"Publ.inst.statist.univ","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2018] Shengyang Sun, Guodong Zhang, Chaoqi Wang, Wenyuan Zeng, Jiaman Li, and Roger Grosse. Dif- ferentiable compositional kernel learning for gaussian pro- cesses. arXiv preprint arXiv:1806.04326, 2018.","title":"Differentiable compositional kernel learning for gaussian processes","authors":"Shengyang Sun, Guodong Zhang, Chaoqi Wang, Wenyuan Zeng, Jiaman Li, Roger Grosse","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Whelan, 2004] Niall Whelan. Sampling from archimedean copulas. Quantitative Finance, 4(3):339-352, 2004.","title":"Sampling from archimedean copulas","authors":"Niall Whelan","year":"2004","venue":"Quantitative Finance","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wilson and Nickisch, 2015] Andrew Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International Conference on Machine Learning, pages 1775-1784, 2015.","title":"Kernel interpolation for scalable structured gaussian processes (kiss-gp)","authors":"Wilson, Nickisch, Andrew Wilson, Hannes Nickisch","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wilson et al., 2016a] Andrew G Wilson, Zhiting Hu, Rus- lan R Salakhutdinov, and Eric P Xing. Stochastic varia- tional deep kernel learning. In Advances in Neural Infor- mation Processing Systems, pages 2586-2594, 2016.","title":"Stochastic variational deep kernel learning","authors":"Wilson","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wilson et al., 2016b] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learn- ing. In Artificial Intelligence and Statistics, pages 370- 378, 2016.","title":"Deep kernel learning","authors":"Wilson","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xue et al., 2019] Hui Xue, Zheng-Fan Wu, and Wei-Xiang Sun. Deep spectral kernel learning. In Proceedings of the Twenty-Eighth International Joint Conference on Ar- tificial Intelligence, IJCAI-19, pages 4019-4025. Interna- tional Joint Conferences on Artificial Intelligence Organi- zation, 7 2019.","title":"Deep spectral kernel learning","authors":"Hui Xue, Zheng-Fan Wu, Wei-Xiang Sun","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/558"},{"position":28,"raw":"[Yaglom, 1987] Akira Moiseevich Yaglom. Correlation Theory of Stationary and Related Random Functions, vol- ume 1. Springer Series in Statistics, 1987.","title":"Akira Moiseevich Yaglom. Correlation Theory of Stationary and Related Random Functions","authors":"Yaglom","year":"1987","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017] Shuai Zhang, Jianxin Li, Pengtao Xie, Yingchun Zhang, Minglai Shao, Haoyi Zhou, and Mengyi Yan. Stacked kernel network. arXiv preprint arXiv:1711.09219, 2017.","title":"Stacked kernel network","authors":"Shuai Zhang, Jianxin Li, Pengtao Xie, Yingchun Zhang, Minglai Shao, Haoyi Zhou, Mengyi Yan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}