{"type":"ScholarlyArticle","title":"Scalable Gaussian Process Regression Networks","authors":[{"name":"Shibo Li","slug":"shibo-li","given":"Shibo","family":"Li","affiliation":"School of Computing , University of Utah"},{"name":"Wei Xing","slug":"wei-xing","given":"Wei","family":"Xing","affiliation":"Scientific Computing and Imaging Institute , University of Utah"},{"name":"Robert M. Kirby","slug":"robert-m-kirby","given":"Robert M.","family":"Kirby","affiliation":"School of Computing , University of Utah; Scientific Computing and Imaging Institute , University of Utah"},{"name":"Shandian Zhe","slug":"shandian-zhe","given":"Shandian","family":"Zhe","affiliation":"School of Computing , University of Utah"}],"abstract":"Gaussian process regression networks (GPRN) are powerful Bayesian models for multi-output regression, but their inference is intractable.  To address this issue, existing methods use a fully factorized structure (or a mixture of such structures) over all the outputs and latent functions for posterior approximation, which, however, can miss the strong posterior dependencies among the latent variables and hurt the inference quality. In addition, the updates of the variational parameters are inefficient and can be prohibitively expensive for a large number of outputs. To overcome these limitations, we propose a scalable variational inference algorithm for GPRN, which not only captures the abundant posterior dependencies but also is much more efficient for massive outputs. We tensorize the output space and introduce tensor/matrix-normal variational posteriors to capture the posterior correlations and to reduce the parameters.\r\nWe jointly optimize all the parameters and exploit the inherent Kronecker product structure in the variational model evidence lower bound to accelerate the computation. We demonstrate the advantages of our method in several real-world applications.","doi":"10.24963/ijcai.2020/340","url":"https://ijcai.org/proceedings/2020/340","pdf":"https://ijcai.org/proceedings/2020/0340.pdf","markdown":"https://ijcai.org/proceedings/2020/340.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/340","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":340,"pages":{"first":2456,"last":2462},"keywords":["Uncertainty in AI: Graphical Models","Uncertainty in AI: Approximate Probabilistic Inference","Machine Learning: Probabilistic Machine Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26694/file_26694.m3u8","length_sec":743,"provider":"hls"}],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learn- ing. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pages 265-283, 2016.","title":"Tensorflow: A system for large-scale machine learning","authors":"Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard","year":"2016","venue":"12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alvarez and Lawrence, 2009] Mauricio Alvarez and Neil D Lawrence. Sparse convolved gaussian processes for multi- output regression. In Advances in neural information pro- cessing systems, pages 57-64, 2009.","title":"Sparse convolved gaussian processes for multioutput regression","authors":"Mauricio Alvarez, Neil D Lawrence","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Álvarez et al., 2010] Mauricio Álvarez, David Luengo, Michalis Titsias, and Neil Lawrence. Efficient multiout- put gaussian processes through variational inducing ker- nels. In Proceedings of the Thirteenth International Con- ference on Artificial Intelligence and Statistics, pages 25- 32, 2010.","title":"Efficient multioutput gaussian processes through variational inducing kernels","authors":"Mauricio Álvarez, David Luengo, Michalis Titsias, Neil Lawrence","year":"2010","venue":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Alvarez et al., 2012] Mauricio A Alvarez, Lorenzo Rosasco, Neil D Lawrence, et al. Kernels for vector- valued functions: A review. Foundations and Trends R in Machine Learning, 4(3):195-266, 2012.","title":"Kernels for vectorvalued functions: A review","authors":"Mauricio A Alvarez, Lorenzo Rosasco, Neil D Lawrence","year":"2012","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Alvarez et al., 2019] Mauricio Alvarez, Wil Ward, and Cris- tian Guarnizo. Non-linear process convolutions for multi- output gaussian processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1969-1977, 2019.","title":"Non-linear process convolutions for multioutput gaussian processes","authors":"Mauricio Alvarez, Wil Ward, Cristian Guarnizo","year":"2019","venue":"The 22nd International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Andreassen et al., 2011] Erik Andreassen, Anders Clausen, Mattias Schevenels, Boyan S Lazarov, and Ole Sigmund. Efficient topology optimization in matlab using 88 lines of code. Structural and Multidisciplinary Optimization, 43(1):1-16, 2011.","title":"Efficient topology optimization in matlab using 88 lines of code","authors":"Andreassen","year":"2011","venue":"Structural and Multidisciplinary Optimization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Balasubramanian and Schwartz, 2002] Mukund Balasubra- manian and Eric L Schwartz. The isomap algorithm and topological stability. Science, 295(5552):7-7, 2002.","title":"Mukund Balasubramanian and Eric L Schwartz. The isomap algorithm and topological stability","authors":"Schwartz Balasubramanian","year":"2002","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bonilla et al., 2007] Edwin V Bonilla, Felix V Agakov, and Christopher KI Williams. Kernel multi-task learning using task-specific features. In Artificial Intelligence and Statis- tics, pages 43-50, 2007.","title":"Kernel multi-task learning using task-specific features","authors":"Felix V Edwin V Bonilla, Agakov, K I Christopher, Williams","year":"2007","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bonilla et al., 2008] Edwin V Bonilla, Kian M Chai, and Christopher Williams. Multi-task gaussian process predic- tion. In Advances in neural information processing sys- tems, pages 153-160, 2008.","title":"Multi-task gaussian process prediction","authors":"Kian M Edwin V Bonilla, Christopher Chai, Williams","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Boyle and Frean, 2005] Phillip Boyle and Marcus Frean. Dependent gaussian processes. In Advances in neural in- formation processing systems, pages 217-224, 2005.","title":"Dependent gaussian processes","authors":"Frean Boyle, Phillip Boyle, Marcus Frean","year":"2005","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Bozeman and Dalton, 1973] James D Bozeman and Charles Dalton. Numerical study of viscous flow in a cavity. Jour- nal of Computational Physics, 12(3):348-363, 1973.","title":"Numerical study of viscous flow in a cavity","authors":"D James, Charles Bozeman, Dalton","year":"1973","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goulard and Voltz, 1992] Michel Goulard and Marc Voltz. Linear coregionalization model: tools for estimation and choice of cross-variogram matrix. Mathematical Geology, 24(3):269-286, 1992.","title":"Linear coregionalization model: tools for estimation and choice of cross-variogram matrix","authors":"Voltz Goulard, Michel Goulard, Marc Voltz","year":"1992","venue":"Mathematical Geology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Higdon et al., 2008] Dave Higdon, James Gattiker, Brian Williams, and Maria Rightley. Computer model calibra- tion using high-dimensional output. Journal of the Ameri- can Statistical Association, 103(482):570-583, 2008.","title":"Computer model calibration using high-dimensional output","authors":"Higdon","year":"2008","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Higdon, 2002] Dave Higdon. Space and space-time model- ing using process convolutions. In Quantitative methods for current environmental issues, pages 37-56. Springer, 2002.","title":"Space and space-time modeling using process convolutions","authors":"Dave Higdon","year":"2002","venue":"Quantitative methods for current environmental issues","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nguyen and Bonilla, 2013] Trung Nguyen and Edwin Bonilla. Efficient variational inference for gaussian process regression networks. In Artificial Intelligence and Statistics, pages 472-480, 2013.","title":"Efficient variational inference for gaussian process regression networks","authors":"Bonilla Nguyen, Trung Nguyen, Edwin Bonilla","year":"2013","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rakitsch et al., 2013] Barbara Rakitsch, Christoph Lippert, Karsten Borgwardt, and Oliver Stegle. It is all in the noise: Efficient multi-task gaussian process inference with struc- tured residuals. In Advances in neural information pro- cessing systems, pages 1466-1474, 2013.","title":"It is all in the noise: Efficient multi-task gaussian process inference with structured residuals","authors":"Barbara Rakitsch, Christoph Lippert, Karsten Borgwardt, Oliver Stegle","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schölkopf et al., 1998] Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear compo- nent analysis as a kernel eigenvalue problem. Neural computation, 10(5):1299-1319, 1998.","title":"Nonlinear component analysis as a kernel eigenvalue problem","authors":"Bernhard Schölkopf, Alexander Smola, Klaus-Robert Müller","year":"1998","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Stegle et al., 2011] Oliver Stegle, Christoph Lippert, Joris M Mooij, Neil D Lawrence, and Karsten Borgwardt. Efficient inference in matrix-variate gaussian models with\\iid observation noise. In Advances in neural information processing systems, pages 630-638, 2011.","title":"Efficient inference in matrix-variate gaussian models with\\iid observation noise","authors":"Oliver Stegle, Christoph Lippert, M Joris, Neil D Mooij, Karsten Lawrence, Borgwardt","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wainwright et al., 2008] Martin J Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends R in Ma- chine Learning, 1(1-2):1-305, 2008.","title":"Graphical models, exponential families, and variational inference","authors":"J Martin, Michael I Wainwright, Jordan","year":"2008","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Williams and Rasmussen, 2006] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for ma- chine learning. MIT press Cambridge, MA, 2006.","title":"Gaussian processes for machine learning","authors":"K I Christopher, Carl Edward Williams, Rasmussen","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wilson et al., 2012] Andrew Gordon Wilson, David A Knowles, and Zoubin Ghahramani. Gaussian process re- gression networks. In Proceedings of the 29th Interna- tional Coference on International Conference on Machine Learning, pages 1139-1146. Omnipress, 2012.","title":"Gaussian process regression networks","authors":"Andrew Gordon, Wilson, David A Knowles, Zoubin Ghahramani","year":"2012","venue":"Proceedings of the 29th International Coference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xing et al., 2015] Wei Xing, Akeel A Shah, and Pras- anth B Nair. Reduced dimensional gaussian process emulators of parametrized partial differential equations based on isomap. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 471(2174):20140697, 2015.","title":"Reduced dimensional gaussian process emulators of parametrized partial differential equations based on isomap","authors":"Wei Xing, Akeel A Shah, Prasanth B Nair","year":"2015","venue":"Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xing et al., 2016] WW Xing, V Triantafyllidis, AA Shah, PB Nair, and Nicholas Zabaras. Manifold learning for the emulation of spatial fields from computational models. Journal of Computational Physics, 326:666-690, 2016.","title":"Manifold learning for the emulation of spatial fields from computational models","authors":"Xing","year":"2016","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhe et al., 2019] Shandian Zhe, Wei Xing, and Robert M Kirby. Scalable high-order gaussian process regression. In The 22nd International Conference on Artificial Intelli- gence and Statistics, pages 2611-2620, 2019.","title":"Scalable high-order gaussian process regression","authors":"Zhe","year":"2019","venue":"The 22nd International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}