{"type":"ScholarlyArticle","title":"Diffusion Variational Autoencoders","authors":[{"name":"Luis A. Perez Rey","slug":"luis-a-perez-rey","given":"Luis A.","family":"Perez Rey","affiliation":"Eindhoven University of Technology , Eindhoven , The Netherlands"},{"name":"Vlado Menkovski","slug":"vlado-menkovski","given":"Vlado","family":"Menkovski","affiliation":"Eindhoven University of Technology , Eindhoven , The Netherlands"},{"name":"Jim Portegies","slug":"jim-portegies","given":"Jim","family":"Portegies","affiliation":"Eindhoven University of Technology , Eindhoven , The Netherlands"}],"abstract":"A standard Variational Autoencoder, with a Euclidean latent space, is structurally incapable of capturing topological properties of certain datasets. To remove topological obstructions, we introduce Diffusion Variational Autoencoders (DeltaVAE) with arbitrary (closed) manifolds as a latent space. A Diffusion Variational Autoencoder uses transition kernels of Brownian motion on the manifold. In particular, it uses properties of the Brownian motion to implement the reparametrization trick and fast approximations to the KL divergence. \r\nWe show that the DeltaVAE is indeed capable of capturing topological properties for datasets with a known underlying latent structure derived from generative processes such as rotations and translations.","doi":"10.24963/ijcai.2020/375","url":"https://ijcai.org/proceedings/2020/375","pdf":"https://ijcai.org/proceedings/2020/0375.pdf","markdown":"https://ijcai.org/proceedings/2020/375.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/375","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":375,"pages":{"first":2704,"last":2710},"keywords":["Machine Learning: Deep Generative Models","Machine Learning: Bayesian Optimization","Machine Learning: Dimensionality Reduction and Manifold Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25231/file_25231.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Arvanitidis et al., 2018] G. Arvanitidis, L. K. Hansen, and S. Hauberg. Latent space oddity: on the curvature of deep generative models. In ICLR, 2018.","title":"Latent space oddity: on the curvature of deep generative models","authors":"G Arvanitidis, L K Hansen, S Hauberg","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berger et al., 1971] M. Berger, P. Gauduchon, and E. Mazet. Le spectre d'une variété riemannienne. Springer, 1971.","title":"Le spectre d'une variété riemannienne","authors":"M Berger, P Gauduchon, E Mazet","year":"1971","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burda et al., 2016] Y. Burda, R. Grosse, and R. Salakhutdi- nov. Importance weighted autoencoders. ICLR, 2016.","title":"Importance weighted autoencoders","authors":"Y Burda, R Grosse, R Salakhutdinov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burgess et al., 2018] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Ler- chner. Understanding disentangling in β-VAE. arXiv:1804.03599, 2018.","title":"Understanding disentangling in β-VAE","authors":"C P Burgess, I Higgins, A Pal, L Matthey, N Watters, G Desjardins, A Lerchner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018] T. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in Variational Autoencoders. In NeurIPS 31. 2018.","title":"Isolating sources of disentanglement in Variational Autoencoders","authors":"T Q Chen, X Li, R B Grosse, D K Duvenaud","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Coifman and Lafon, 2006] R. R. Coifman and S. Lafon. Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5-30, jul 2006.","title":"Diffusion maps","authors":"R R Coifman, S Lafon","year":"2006","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Davidson et al., 2018] T. R. Davidson, L. Falorsi, N. De Cao, T. Kipf, and J. M. Tomczak. Hyper- spherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence. 2018.","title":"Hyperspherical variational auto-encoders","authors":"T R Davidson, L Falorsi, N De Cao, T Kipf, J M Tomczak","year":"2018","venue":"34th Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[de Haan and Falorsi, 2018] P. de Haan and L. Falorsi. Topological constraints on homeomorphic auto-encoding. arXiv preprint, arXiv:1812.10783, 2018.","title":"Topological constraints on homeomorphic auto-encoding","authors":"P De Haan, L Falorsi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dilokthanakul et al., 2016] N. Dilokthanakul, P. A. M. Me- diano, M. Garnelo, M. C. H. Lee, H. Salimbeni, K. Arulku- maran, and M. Shanahan. Deep unsupervised clustering with Gaussian mixture Variational Autoencoders. arXiv preprint, arXiv:1611.02648, 2016.","title":"Deep unsupervised clustering with Gaussian mixture Variational Autoencoders","authors":"N Dilokthanakul, P A M Mediano, M Garnelo, M C H Lee, H Salimbeni, K Arulkumaran, M Shanahan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Falorsi et al., 2018] L. Falorsi, P. de Haan, T. R. David- son, N. De Cao, M. Weiler, P. Forré, and T. S. Cohen. Explorations in homeomorphic variational auto-encoding. arXiv:1807.04689, 2018.","title":"Explorations in homeomorphic variational auto-encoding","authors":"L Falorsi, P De Haan, T R Davidson, N De Cao, M Weiler, P Forré, T S Cohen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Figurnov et al., 2018] M. Figurnov, S. Mohamed, and A. Mnih. Implicit Reparameterization Gradients. NeurIPS, 2018.","title":"Implicit Reparameterization Gradients","authors":"M Figurnov, S Mohamed, A Mnih","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Higgins et al., 2017] I.","title":null,"authors":"Higgins","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. β-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2017.","title":"β-vae: Learning basic visual concepts with a constrained variational framework","authors":"L Higgins, A Matthey, C Pal, X Burgess, M Glorot, S Botvinick, A Mohamed, Lerchner","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Higgins et al., 2018] I.","title":null,"authors":"Higgins","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerch- ner. Towards a definition of disentangled representations. arXiv:1812.02230, 2018.","title":"Towards a definition of disentangled representations","authors":"D Higgins, D Amos, S Pfau, L Racaniere, D Matthey, A Rezende, Lerchner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hinton and Zemel, 1994] G. E. Hinton and R. S. Zemel. Autoencoders, minimum description length and helmholtz free energy. In NeurIPS, 1994.","title":"Autoencoders, minimum description length and helmholtz free energy","authors":"G E Hinton, R S Zemel","year":"1994","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Honkela and Valpola, 2004] A. Honkela and H. Valpola. Variational learning and bits-back coding: An information-theoretic view to Bayesian learning. IEEE Transactions on Neural Networks, 15(4):800-810, 7 2004.","title":"Variational learning and bits-back coding: An information-theoretic view to Bayesian learning","authors":"A Honkela, H Valpola","year":"2004","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hsu, 2008] E. P. Hsu. A brief introduction to Brownian mo- tion on a Riemannian manifold. Lecture notes, 2008.","title":"A brief introduction to Brownian motion on a Riemannian manifold","authors":"E P Hsu","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jørgensen, 1975] E. Jørgensen. The central limit problem for geodesic random walks. Zeitschrift für Wahrschein- lichkeitstheorie und verwandte Gebiete, 32(1-2):1-64, 1975.","title":"The central limit problem for geodesic random walks","authors":"E Jørgensen","year":"1975","venue":"Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kim and Mnih, 2018] H. Kim and A. Mnih. Disentangling by factorising. arXiv:1802.05983, 2018.","title":"Disentangling by factorising","authors":"H Kim, A Mnih","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kingma and Welling, 2014] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In ICLR, 2014.","title":"Auto-Encoding Variational Bayes","authors":"D P Kingma, M Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kumar et al., 2018] A. Kumar, P. Sattigeri, and A. Balakr- ishnan. Variational inference of distentangled latent con- cepts from unlabeled observations. ICLR, 2018.","title":"Variational inference of distentangled latent concepts from unlabeled observations","authors":"A Kumar, P Sattigeri, A Balakrishnan","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu and Zhu, 2017] C. Liu and J. Zhu. Riemannian Stein variational gradient descent for Bayesian inference. arXiv:1711.11216, 2017.","title":"Riemannian Stein variational gradient descent for Bayesian inference","authors":"C Liu, J Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mathieu et al., 2019] E. Mathieu, C. L. Lan, C. J. Maddi- son, R. Tomioka, and Y. W. Teh. Hierarchical Representa- tions with Poincaré Variational Auto-Encoders. NeurIPS, 2019.","title":"Hierarchical Representations with Poincaré Variational Auto-Encoders","authors":"E Mathieu, C L Lan, C J Maddison, R Tomioka, Y W Teh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nagano et al., 2019] Y. Nagano, S. Yamaguchi, Y. Fujita, and M. Koyama. A Wrapped Normal Distribution on Hy- perbolic Space for Gradient-Based Learning. ICML, 2019.","title":"A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning","authors":"Y Nagano, S Yamaguchi, Y Fujita, M Koyama","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Portegies, 2018] J. W. Portegies. Ergo learning. Nieuw Archief voor Wiskunde, 5(3):199-205, 2018.","title":"Ergo learning","authors":"J W Portegies","year":"2018","venue":"Nieuw Archief voor Wiskunde","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rezende et al., 2014] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv:1401.4082, 2014.","title":"Stochastic backpropagation and approximate inference in deep generative models","authors":"D J Rezende, S Mohamed, D Wierstra","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rissanen, 1978] J. Rissanen. Modeling by shortest data de- scription. Automatica, 14(5):465-471, 9 1978.","title":"Modeling by shortest data description","authors":"J Rissanen","year":"1978","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Roweis, 2000] S. T. Roweis. Nonlinear Dimensional- ity Reduction by Locally Linear Embedding. Science, 290(5500):2323-2326, dec 2000.","title":"Nonlinear Dimensionality Reduction by Locally Linear Embedding","authors":"S T Roweis","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Salakhutdinov and Murray, 2008] R. Salakhutdinov and I. Murray. On the quantitative analysis of deep belief networks. In ICML, 2008.","title":"On the quantitative analysis of deep belief networks","authors":"R Salakhutdinov, I Murray","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Salimans et al., 2015] T. Salimans, D. Kingma, and M. Welling. Markov Chain Monte Carlo and Variational Inference: Bridging the gap. In ICML, 2015.","title":"Markov Chain Monte Carlo and Variational Inference: Bridging the gap","authors":"T Salimans, D Kingma, M Welling","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Schmidhuber, 1997] J. Schmidhuber. Discovering neural nets with low Kolmogorov complexity and high gener- alization capability. Neural Networks, 10(5):857 -873, 1997.","title":"Discovering neural nets with low Kolmogorov complexity and high generalization capability","authors":"J Schmidhuber","year":"1997","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Solomonoff, 1964] R.J. Solomonoff. A formal theory of in- ductive inference. Part I. Information and Control, 7(1):1- 22, 3 1964.","title":"A formal theory of inductive inference","authors":"R J Solomonoff","year":"1964","venue":"Part I. Information and Control","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tenenbaum et al., 2000] J. B. Tenenbaum, V. De Silva, and J. C. Langford. A global geometric framework for nonlin- ear dimensionality reduction. Science, 290(5500):2319- 2323, 2000.","title":"A global geometric framework for nonlinear dimensionality reduction","authors":"J B Tenenbaum, V De Silva, J C Langford","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tomczak and Welling, 2017] J. M. Tomczak and M. Welling. VAE with a VampPrior. arXiv:1705.07120, 2017.","title":"VAE with a VampPrior","authors":"J M Tomczak, M Welling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tosi et al., 2014] A. Tosi, S. Hauberg, A. Vellido, and N. D. Lawrence. Metrics for probabilistic geometries. In UAI, 2014.","title":"Metrics for probabilistic geometries","authors":"A Tosi, S Hauberg, A Vellido, N D Lawrence","year":"2014","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wu et al., 2014] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D ShapeNets: A Deep Representation for Volumetric Shapes. 2014.","title":"3D ShapeNets: A Deep Representation for Volumetric Shapes","authors":"Z Wu, S Song, A Khosla, F Yu, L Zhang, X Tang, J Xiao","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}