{"type":"ScholarlyArticle","title":"Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry (Extended Abstract)","authors":[{"name":"Jonas Gregor Wiese","slug":"jonas-gregor-wiese","given":"Jonas Gregor","family":"Wiese","affiliation":"Technical University of Munich , Germany"},{"name":"Lisa Wimmer","slug":"lisa-wimmer","given":"Lisa","family":"Wimmer","affiliation":"Department of Statistics , LMU Munich , Germany; Munich Center for Machine Learning (MCML) , Germany"},{"name":"Theodore Papamarkou","slug":"theodore-papamarkou","given":"Theodore","family":"Papamarkou","affiliation":"Department of Mathematics , The University of Manchester , UK"},{"name":"Bernd Bischl","slug":"bernd-bischl","given":"Bernd","family":"Bischl","affiliation":"Department of Statistics , LMU Munich , Germany; Munich Center for Machine Learning (MCML) , Germany"},{"name":"Stephan Guennemann","slug":"stephan-guennemann","given":"Stephan","family":"Guennemann","affiliation":"Technical University of Munich , Germany; Munich Center for Machine Learning (MCML) , Germany"},{"name":"David Ruegamer","slug":"david-ruegamer","given":"David","family":"Ruegamer","affiliation":"Department of Statistics , LMU Munich , Germany; Munich Center for Machine Learning (MCML) , Germany"}],"abstract":"Bayesian inference in deep neural networks is challenging due to the high-dimensional, strongly multi-modal parameter posterior density landscape. Markov chain Monte Carlo approaches asymptotically recover the true posterior but are considered prohibitively expensive for large modern architectures. We argue that the dilemma between exact-but-unaffordable and cheap-but-inexact approaches can be mitigated by exploiting symmetries in the posterior landscape. We show theoretically that the posterior predictive density in Bayesian neural networks can be restricted to a symmetry-free parameter reference set. By further deriving an upper bound on the number of Monte Carlo chains required to capture the functional diversity, we propose a straightforward approach for feasible Bayesian inference.","doi":"10.24963/ijcai.2024/943","url":"https://ijcai.org/proceedings/2024/943","pdf":"https://ijcai.org/proceedings/2024/0943.pdf","markdown":"https://ijcai.org/proceedings/2024/943.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/943","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":943,"pages":{"first":8466,"last":8470},"keywords":["Uncertainty in AI: UAI: Tractable probabilistic models","Uncertainty in AI: UAI: Inference","Machine Learning: ML: Bayesian learning","Machine Learning: ML: Probabilistic machine learning"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 1993] An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the Geometry of Feedforward Neural Network Error Surfaces. Neural Computation, 5(6):910- 927, 1993.","title":"On the Geometry of Feedforward Neural Network Error Surfaces","authors":"Chen","year":"1993","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Daxberger et al., 2021] Erik Daxberger, Agustinus Kris- tiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace Redux -Effortless Bayesian Deep Learning. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021), 2021.","title":"Laplace Redux -Effortless Bayesian Deep Learning","authors":"Daxberger","year":"2021","venue":"th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dua and Graff, 2017] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.","title":"Dheeru Dua and Casey Graff. UCI machine learning repository","authors":"Graff Dua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gelman et al., 2014] Andrew Gelman, Jessica Hwang, and Aki Vehtari. Understanding predictive information cri- teria for Bayesian models. Statistics and Computing, 24(6):997-1016, 2014.","title":"Understanding predictive information criteria for Bayesian models","authors":"Andrew Gelman, Jessica Hwang, Aki Vehtari","year":"2014","venue":"Statistics and Computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hecht-Nielsen, 1990] Robert Hecht-Nielsen. On the alge- braic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129-135. Elsevier, 1990.","title":"On the algebraic structure of feedforward network weight spaces","authors":"Robert Hecht-Nielsen","year":"1990","venue":"Advanced Neural Computers","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hoffman and Gelman, 2014] Matthew D. Hoffman and An- drew Gelman. The No-U-Turn Sampler: Adaptively Set- ting Path Lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(47):1593-1623, 2014.","title":"The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo","authors":"Matthew D Hoffman, Andrew Gelman","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hüllermeier and Waegeman, 2021] Eyke Hüllermeier and Willem Waegeman. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods. Machine Learning, 2021.","title":"Eyke Hüllermeier and Willem Waegeman. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods","authors":"Waegeman Hüllermeier","year":"2021","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Izmailov et al., 2021] Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What Are Bayesian Neural Network Posteriors Really Like? In Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021.","title":"What Are Bayesian Neural Network Posteriors Really Like","authors":"Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, Andrew Gordon, Wilson","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kůrková and Kainen, 1994] Věra Kůrková and Paul C. Kainen. Functionally Equivalent Feedforward Neural Net- works. Neural Computation, 6(3):543-558, 1994.","title":"Functionally Equivalent Feedforward Neural Networks","authors":"Kainen ; ] Věra Kůrková, Paul C Kůrková, Kainen","year":"1994","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lakshminarayanan et al., 2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scal- able Predictive Uncertainty Estimation using Deep Ensem- bles. In Proceedings of the 31st Conference on Neural In- formation Processing Systems (NIPS 2017), 2017.","title":"Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles","authors":"Alexander Balaji Lakshminarayanan, Charles Pritzel, Blundell","year":"2017","venue":"Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[MacKay, 1992] David J. C. MacKay. Bayesian Interpola- tion. Neural Computation, 4:415-447, 1992.","title":"Bayesian Interpolation","authors":"J C David, Mackay","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Petzka et al., 2020] Henning Petzka, Martin Trimmel, and Cristian Sminchisescu. Notes on the Symmetries of 2- Layer ReLU-Networks. Northern Lights Deep Learning Workshop, 1, 2020.","title":"Notes on the Symmetries of 2-Layer ReLU-Networks","authors":"Henning Petzka, Martin Trimmel, Cristian Sminchisescu","year":"2020","venue":"Northern Lights Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sommer et al., 2024] Emanuel Sommer, Lisa Wimmer, Theodore Papamarkou, Ludwig Bothmann, Bernd Bischl, and David Rügamer. Connecting the Dots: Is Mode- Connectedness the Key to Feasible Sample-Based Infer- ence in Bayesian Neural Networks? In Proceedings of the 41th International Conference on Machine Learning. PMLR, 2024.","title":"Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks","authors":"Sommer","year":"2024","venue":"Proceedings of the 41th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sussmann, 1992] Héctor J. Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input- output map. Neural Networks, 5(4):589-593, 1992.","title":"Uniqueness of the weights for minimal feedforward nets with a given inputoutput map","authors":"J Héctor, Sussmann","year":"1992","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vlačić and Bölcskei, 2021] Verner Vlačić and Helmut Bölcskei. Affine symmetries and neural network iden- tifiability. Advances in Mathematics, 376:107485, 2021.","title":"Affine symmetries and neural network identifiability","authors":"Bölcskei Vlačić, Verner Vlačić, Helmut Bölcskei","year":"2021","venue":"Advances in Mathematics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}