{"type":"ScholarlyArticle","title":"Variational Learning of Bayesian Neural Networks via Bayesian Dark Knowledge","authors":[{"name":"Gehui Shen","slug":"gehui-shen","given":"Gehui","family":"Shen","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University"},{"name":"Xi Chen","slug":"xi-chen","given":"Xi","family":"Chen","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University"},{"name":"Zhihong Deng","slug":"zhihong-deng","given":"Zhihong","family":"Deng","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University"}],"abstract":"Bayesian neural networks (BNNs) have received more and more attention because they are capable of modeling epistemic uncertainty which is hard for conventional neural networks. Markov chain Monte Carlo (MCMC) methods and variational inference (VI) are two mainstream methods for Bayesian deep learning. The former is effective but its storage cost is prohibitive since it has to save many samples of neural network parameters. The latter method is more time and space efficient, however the approximate variational posterior limits its performance. In this paper, we aim to combine the advantages of above two methods by distilling MCMC samples into an approximate variational posterior. On the basis of an existing distillation technique we first propose variational Bayesian dark knowledge method. Moreover, we propose Bayesian dark prior knowledge, a novel distillation method which considers MCMC posterior as the prior of a variational BNN. Two proposed methods both not only can reduce the space overhead of the teacher model so that are scalable, but also maintain a distilled posterior distribution capable of modeling epistemic uncertainty. Experimental results manifest our methods outperform existing distillation method in terms of predictive accuracy and uncertainty modeling.","doi":"10.24963/ijcai.2020/282","url":"https://ijcai.org/proceedings/2020/282","pdf":"https://ijcai.org/proceedings/2020/0282.pdf","markdown":"https://ijcai.org/proceedings/2020/282.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/282","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":282,"pages":{"first":2037,"last":2043},"keywords":["Machine Learning: Deep Learning","Uncertainty in AI: Approximate Probabilistic Inference","Machine Learning: Probabilistic Machine Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24351/file_24351.m3u8","length_sec":290,"provider":"hls"}],"references":[{"position":1,"raw":"[Balan et al., 2015] Anoop Korattikara Balan, Vivek Rathod, Kevin P. Murphy, and Max Welling. Bayesian dark knowl- edge. In Proceedings of NIPS, pages 3438-3446, 2015.","title":"Bayesian dark knowledge","authors":"Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, Max Welling","year":"2015","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blundell et al., 2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncer- tainty in neural network. In Proceedings of ICML, pages 1613-1622, 2015.","title":"Weight uncertainty in neural network","authors":"Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, Daan Wierstra","year":"2015","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2014] Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Proceedings of ICML, pages 1683-1691, 2014.","title":"Stochastic gradient hamiltonian monte carlo","authors":"Tianqi Chen, Emily B Fox, Carlos Guestrin","year":"2014","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Depeweg et al., 2017] Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. In Proceedings of ICLR, 2017.","title":"Learning and policy search in stochastic dynamical systems with bayesian neural networks","authors":"Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, Steffen Udluft","year":"2017","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ding et al., 2014] Nan Ding, Youhan Fang, Ryan Babbush, Changyou Chen, Robert D. Skeel, and Hartmut Neven. Bayesian sampling using stochastic gradient thermostats. In Proceedings of NIPS, pages 3203-3211, 2014.","title":"Bayesian sampling using stochastic gradient thermostats","authors":"Nan Ding, Youhan Fang, Ryan Babbush, Changyou Chen, Robert D Skeel, Hartmut Neven","year":"2014","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gal and Ghahramani, 2016a] Yarin Gal and Zoubin Ghahra- mani. Bayesian convolutional neural networks with bernoulli approximate variational inference. In ICLR, Work- shop Track Proceedings, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approximate variational inference","authors":"Ghahramani Gal","year":"2016","venue":"ICLR, Workshop Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gal and Ghahramani, 2016b] Yarin Gal and Zoubin Ghahra- mani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of ICML, pages 1050-1059, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Ghahramani Gal","year":"2016","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gal et al., 2017] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of ICML, pages 1183-1192, 2017.","title":"Deep bayesian active learning with image data","authors":"Yarin Gal, Riashat Islam, Zoubin Ghahramani","year":"2017","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gal, 2016] Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.","title":"Uncertainty in Deep Learning","authors":"Yarin Gal","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron C. Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Proceedings of NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, Yoshua Bengio","year":"2014","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodfellow et al., 2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proceedings of ICLR, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goodfellow et al., 2016] Ian J. Goodfellow, Nicolas Paper- not, and Patrick D. McDaniel. cleverhans v0.1: an adver- sarial machine learning library. CoRR, abs/1610.00768, 2016.","title":"cleverhans v0.1: an adversarial machine learning library","authors":"Ian J Goodfellow, Nicolas Papernot, Patrick D Mcdaniel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Graves, 2011] Alex Graves. Practical variational inference for neural networks. In Proceedings of NIPS, pages 2348- 2356, 2011.","title":"Practical variational inference for neural networks","authors":"Alex Graves","year":"2011","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Guo et al., 2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kil- ian Q. Weinberger. On calibration of modern neural net- works. In Proceedings of ICML, pages 1321-1330, 2017.","title":"On calibration of modern neural networks","authors":"Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q Weinberger","year":"2017","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hinton and van Camp, 1993] Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimiz- ing the description length of the weights. In Proceedings of COLT, pages 5-13, 1993.","title":"Keeping the neural networks simple by minimizing the description length of the weights","authors":"Geoffrey E Hinton, Drew Van Camp","year":"1993","venue":"Proceedings of COLT","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hinton et al., 2014] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop, 2014.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey E Hinton, Oriol Vinyals, Jeffrey Dean","year":"2014","venue":"NIPS Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Houlsby et al., 2011] Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learn- ing for classification and preference learning. CoRR, abs/1112.5745, 2011.","title":"Bayesian active learning for classification and preference learning","authors":"Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, Máté Lengyel","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kendall and Gal, 2017] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Proceedings of NIPS, pages 5574- 5584, 2017.","title":"What uncertainties do we need in bayesian deep learning for computer vision","authors":"Alex Kendall, Yarin Gal","year":"2017","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Louizos and Welling, 2016] Christos Louizos and Max Welling. Structured and efficient variational deep learn- ing with matrix gaussian posteriors. In Proceedings of ICML, pages 1708-1716, 2016.","title":"Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors","authors":"Welling Louizos","year":"2016","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Louizos and Welling, 2017] Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of ICML, pages 2218-2227, 2017.","title":"Multiplicative normalizing flows for variational bayesian neural networks","authors":"Louizos, Welling ; ], Christos Louizos, Max Welling","year":"2017","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[MacKay, 1992] David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural Compu- tation, 4(3):448-472, 1992.","title":"A practical bayesian framework for backpropagation networks","authors":"J C David, Mackay","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Neal, 1995] GRadford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.","title":"Bayesian learning for neural networks","authors":"M Gradford, Neal","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pawlowski et al., 2017] Nick Pawlowski, Miguel Jaques, and Ben Glocker. Efficient variational bayesian neural net- work ensembles for outlier detection. In ICLR, Workshop Track Proceedings, 2017.","title":"Efficient variational bayesian neural network ensembles for outlier detection","authors":"Nick Pawlowski, Miguel Jaques, Ben Glocker","year":"2017","venue":"ICLR, Workshop Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Siddhant and Lipton, 2018] Aditya Siddhant and Zachary C. Lipton. Deep bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of EMNLP, pages 2904-2909, 2018.","title":"Deep bayesian active learning for natural language processing: Results of a large-scale empirical study","authors":"Aditya Siddhant, Zachary C Lipton","year":"2018","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Szegedy et al., 2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of ICLR, 2014.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2014","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2018] Kuan-Chieh Wang, Paul Vicol, James Lu- cas, Li Gu, Roger Grosse, and Richard S. Zemel. Adver- sarial distillation of bayesian neural network posteriors. In Proceedings of ICML, pages 5177-5186, 2018.","title":"Adversarial distillation of bayesian neural network posteriors","authors":"Kuan-Chieh Wang, Paul Vicol, James Lucas, Li Gu, Roger Grosse, Richard S Zemel","year":"2018","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Welling and Teh, 2011] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynam- ics. In Proceedings of ICML, pages 681-688, 2011.","title":"Bayesian learning via stochastic gradient langevin dynamics","authors":"Teh ; ] Max Welling, Yee Whye Welling, Teh","year":"2011","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}