{"type":"ScholarlyArticle","title":"Deeper Connections between Neural Networks and Gaussian Processes Speed-up Active Learning","authors":[{"name":"Evgenii Tsymbalov","slug":"evgenii-tsymbalov","given":"Evgenii","family":"Tsymbalov","affiliation":"Skolkovo Institute of Science and Technology (Skoltech)"},{"name":"Sergei Makarychev","slug":"sergei-makarychev","given":"Sergei","family":"Makarychev","affiliation":"Skolkovo Institute of Science and Technology (Skoltech)"},{"name":"Alexander Shapeev","slug":"alexander-shapeev","given":"Alexander","family":"Shapeev","affiliation":"Skolkovo Institute of Science and Technology (Skoltech)"},{"name":"Maxim Panov","slug":"maxim-panov","given":"Maxim","family":"Panov","affiliation":"Skolkovo Institute of Science and Technology (Skoltech)"}],"abstract":"Active learning methods for neural networks are usually based on greedy criteria, which ultimately give a single new design point for the evaluation. Such an approach requires either some heuristics to sample a batch of design points at one active learning iteration, or retraining the neural network after adding each data point, which is computationally inefficient. Moreover, uncertainty estimates for neural networks sometimes are overconfident for the points lying far from the training sample.  In this work, we propose to approximate Bayesian neural networks (BNN) by Gaussian processes (GP), which allows us to update the uncertainty estimates of predictions efficiently without retraining the neural network while avoiding overconfident uncertainty prediction for out-of-sample points. In a series of experiments on real-world data, including large-scale problems of chemical and physical modeling, we show the superiority of the proposed approach over the state-of-the-art methods.","doi":"10.24963/ijcai.2019/499","url":"https://ijcai.org/proceedings/2019/499","pdf":"https://ijcai.org/proceedings/2019/0499.pdf","markdown":"https://ijcai.org/proceedings/2019/499.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/499","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":499,"pages":{"first":3599,"last":3605},"keywords":["Uncertainty in AI: Approximate Probabilistic Inference","Machine Learning: Active Learning","Uncertainty in AI: Bayesian Networks","Machine Learning: Deep Learning","Uncertainty in AI: Uncertainty in AI"],"videos":[],"references":[{"position":1,"raw":"[Beluch et al., 2018] Beluch, W. H., Genewein, T., Nürnberger, A., and Köhler, J. M. (2018). The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9368-9377.","title":"The power of ensembles for active learning in image classification","authors":"W H Beluch, T Genewein, A Nürnberger, J M Köhler","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertin-Mahieux et al., 2011] Bertin-Mahieux, T., Ellis, D. P., Whitman, B., and Lamere, P. (2011). The million song dataset. In Ismir, volume 2, page 10.","title":"The million song dataset","authors":"T Bertin-Mahieux, D P Ellis, B Whitman, P Lamere","year":"2011","venue":"Ismir","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bradshaw et al., 2017] Bradshaw, J., Matthews, A. G. d. G., and Ghahramani, Z. (2017). Adversarial examples, uncer- tainty, and transfer testing robustness in gaussian process hybrid deep networks. arXiv:1707.02476.","title":"Adversarial examples, uncer-tainty, and transfer testing robustness in gaussian process hybrid deep networks","authors":"J Bradshaw, A G D G Matthews, Z Ghahramani","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burnaev and Panov, 2015] Burnaev, E. and Panov, M. (2015). Adaptive design of experiments based on gaus- sian processes. In International Symposium on Statistical Learning and Data Sciences, pages 116-125. Springer.","title":"Adaptive design of experiments based on gaussian processes","authors":"E Burnaev, M Panov","year":"2015","venue":"International Symposium on Statistical Learning and Data Sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Buza, 2014] Buza, K. (2014). Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery, pages 145-152. Springer.","title":"Feedback prediction for blogs","authors":"K Buza","year":"2014","venue":"Data analysis, machine learning and knowledge discovery","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fernandes et al., 2015] Fernandes, K., Vinagre, P., and Cortez, P. (2015). A proactive intelligent decision sup- port system for predicting the popularity of online news. In Portuguese Conference on Artificial Intelligence, pages 535-546. Springer.","title":"A proactive intelligent decision support system for predicting the popularity of online news","authors":"K Fernandes, P Vinagre, P Cortez","year":"2015","venue":"Portuguese Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Forrester et al., 2008] Forrester, A., Keane, A., et al. (2008). Engineering design via surrogate modelling: a practical guide. John Wiley & Sons.","title":"Engineering design via surrogate modelling: a practical guide","authors":"A Forrester, A Keane","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gal and Ghahramani, 2016] Gal, Y. and Ghahramani, Z. (2016). Dropout as a bayesian approximation: Represent- ing model uncertainty in deep learning. In Proc. ICML'16, pages 1050-1059.","title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Y Gal, Z Ghahramani","year":"2016","venue":"Proc. ICML'16","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gal et al., 2017] Gal, Y., Islam, R., and Ghahramani, Z. (2017). Deep bayesian active learning with image data. arXiv:1703.02910.","title":"Deep bayesian active learning with image data","authors":"Y Gal, R Islam, Z Ghahramani","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garnelo et al., 2018] Garnelo, M., Rosenbaum, D., Maddi- son, C. J., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D. J., and Eslami, S. (2018). Conditional neural processes. arXiv:1807.01613.","title":"Conditional neural processes","authors":"M Garnelo, D Rosenbaum, C J Maddison, T Ramalho, D Saxton, M Shanahan, Y W Teh, D J Rezende, S Eslami","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Graf et al., 2011] Graf, F., Kriegel, H.-P., Schubert, M., Pölsterl, S., and Cavallaro, A. (2011). 2d image regis- tration in ct images using radial image descriptors. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 607-614. Springer.","title":"2d image registration in ct images using radial image descriptors","authors":"F Graf, H.-P Kriegel, M Schubert, S Pölsterl, A Cavallaro","year":"2011","venue":"International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Graves, 2011] Graves, A. (2011). Practical variational in- ference for neural networks. In Advances in neural infor- mation processing systems, pages 2348-2356.","title":"Practical variational inference for neural networks","authors":"A Graves","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hafner et al., 2018] Hafner, D., Tran, D., Irpan, A., Lilli- crap, T., and Davidson, J. (2018). Reliable uncertainty estimates in deep neural networks using noise contrastive priors. arXiv:1807.09289.","title":"Reliable uncertainty estimates in deep neural networks using noise contrastive priors","authors":"D Hafner, D Tran, A Irpan, T Lillicrap, J Davidson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hensman et al., 2013] Hensman, J., Fusi, N., and Lawrence, N. D. (2013). Gaussian processes for big data. arXiv:1309.6835.","title":"Gaussian processes for big data","authors":"J Hensman, N Fusi, N D Lawrence","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2017] Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J. (2017). Deep neural networks as gaussian processes. arXiv:1711.00165.","title":"Deep neural networks as gaussian processes","authors":"J Lee, Y Bahri, R Novak, S S Schoenholz, J Pennington, J Sohl-Dickstein","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018] Li, H., Wang, X., and Ding, S. (2018). Re- search and development of neural network ensembles: a survey. Artificial Intelligence Review, 49(4):455-479.","title":"Research and development of neural network ensembles: a survey","authors":"H Li, X Wang, S Ding","year":"2018","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Matthews et al., 2018] Matthews, A. G. d. G., Rowland, M., Hron, J., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. arXiv:1804.11271. [Mentch and Hooker, 2016]","title":"Gaussian process behaviour in wide deep neural networks","authors":"A G D G Matthews, M Rowland, J Hron, R E Turner, Z Ghahramani","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Mentch, L. and Hooker, G. (2016). Quantifying uncertainty in random forests via con- fidence intervals and hypothesis tests. The Journal of Ma- chine Learning Research, 17(1):841-881.","title":"Quantifying uncertainty in random forests via confidence intervals and hypothesis tests","authors":"L Mentch, G Hooker","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nugteren and Codreanu, 2015] Nugteren, C. and Codreanu, V. (2015). Cltune: A generic auto-tuner for opencl ker- nels. In Embedded Multicore/Many-core Systems-on-Chip (MCSoC), 2015 IEEE 9th International Symposium on, pages 195-202. IEEE.","title":"Cltune: A generic auto-tuner for opencl kernels","authors":"C Nugteren, V Codreanu","year":"2015","venue":"Embedded Multicore/Many-core Systems-on-Chip (MCSoC)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Paisley et al., 2012] Paisley, J., Blei, D. M., and Jordan, M. I. (2012). Variational bayesian inference with stochas- tic search. In Proc. ICML'12, pages 1363-1370.","title":"Variational bayesian inference with stochastic search","authors":"J Paisley, D M Blei, M I Jordan","year":"2012","venue":"Proc. ICML'12","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Podryabinkin et al., 2013] Podryabinkin, E., Rudyak, V., Gavrilov, A., and May, R. (2013). Detailed modeling of drilling fluid flow in a wellbore annulus while drilling. In Proceedings of the International Conference on Offshore Mechanics and Arctic Engineering -OMAE, volume 6.","title":"Detailed modeling of drilling fluid flow in a wellbore annulus while drilling","authors":"E Podryabinkin, V Rudyak, A Gavrilov, R May","year":"2013","venue":"Proceedings of the International Conference on Offshore Mechanics and Arctic Engineering -OMAE","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pop and Fulop, 2018] Pop, R. and Fulop, P. (2018). Deep ensemble bayesian active learning: Addressing the mode collapse issue in monte carlo dropout via ensembles. arXiv preprint arXiv:1811.03897.","title":"Deep ensemble bayesian active learning: Addressing the mode collapse issue in monte carlo dropout via ensembles","authors":"Pop, R Fulop ; Pop, P Fulop","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ramakrishnan et al., 2014] Ramakrishnan, R., Dral, P. O., Rupp, M., and Von Lilienfeld, O. A. (2014). Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1:140022.","title":"Quantum chemistry structures and properties of 134 kilo molecules","authors":"R Ramakrishnan, P O Dral, M Rupp, Von Lilienfeld, O A","year":"2014","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rasmussen, 2004] Rasmussen, C. E. (2004). Gaussian pro- cesses in machine learning. In Advanced lectures on ma- chine learning, pages 63-71. Springer.","title":"Gaussian processes in machine learning","authors":"C E Rasmussen","year":"2004","venue":"Advanced lectures on machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sacks et al., 1989] Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). Design and analysis of computer experiments. Statistical science, pages 409-423.","title":"Design and analysis of computer experiments","authors":"J Sacks, W J Welch, T J Mitchell, H P Wynn","year":"1989","venue":"Statistical science","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schütt et al., 2017] Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., and Tkatchenko, A. (2017). Quantum- chemical insights from deep tensor neural networks. Na- ture communications, 8:13890.","title":"Quantumchemical insights from deep tensor neural networks","authors":"K T Schütt, F Arbabzadah, S Chmiela, K R Müller, A Tkatchenko","year":"2017","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Settles, 2012] Settles, B. (2012). Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1-114.","title":"Active learning","authors":"B Settles","year":"2012","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Seung et al., 1992] Seung, H. S., Opper, M., and Sompolin- sky, H. (1992). Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pages 287-294. ACM.","title":"Query by committee","authors":"H S Seung, M Opper, H Sompolinsky","year":"1992","venue":"Proceedings of the fifth annual workshop on Computational learning theory","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shannon et al., 2003] Shannon, P., Markiel, A., Ozier, O., Baliga, N. S., Wang, J. T., Ramage, D., Amin, N., Schwikowski, B., and Ideker, T. (2003). Cytoscape: a software environment for integrated models of biomolecu- lar interaction networks. Genome research, 13(11):2498- 2504.","title":"Cytoscape: a software environment for integrated models of biomolecular interaction networks","authors":"P Shannon, A Markiel, O Ozier, N S Baliga, J T Wang, D Ramage, N Amin, B Schwikowski, T Ideker","year":"2003","venue":"Genome research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sun et al., 2018] Sun, S., Zhang, G., Wang, C., Zeng, W., Li, J., and Grosse, R. (2018). Differentiable compositional kernel learning for gaussian processes. arXiv:1806.04326.","title":"Differentiable compositional kernel learning for gaussian processes","authors":"S Sun, G Zhang, C Wang, W Zeng, J Li, R Grosse","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tsymbalov et al., 2018] Tsymbalov, E., Panov, M., and Shapeev, A. (2018). Dropout-based active learning for re- gression. In International Conference on Analysis of Im- ages, Social Networks and Texts, pages 247-258. Springer.","title":"Dropout-based active learning for regression","authors":"E Tsymbalov, M Panov, A Shapeev","year":"2018","venue":"International Conference on Analysis of Images, Social Networks and Texts","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}