{"type":"ScholarlyArticle","title":"Student-t Variational Autoencoder for Robust Density Estimation","authors":[{"name":"Hiroshi Takahashi","slug":"hiroshi-takahashi","given":"Hiroshi","family":"Takahashi","affiliation":"NTT Software Innovation Center"},{"name":"Tomoharu Iwata","slug":"tomoharu-iwata","given":"Tomoharu","family":"Iwata","affiliation":"NTT Communication Science Laboratories"},{"name":"Yuki Yamanaka","slug":"yuki-yamanaka","given":"Yuki","family":"Yamanaka","affiliation":"NTT Secure Platform Laboratories"},{"name":"Masanori Yamada","slug":"masanori-yamada","given":"Masanori","family":"Yamada","affiliation":"NTT Secure Platform Laboratories"},{"name":"Satoshi Yagi","slug":"satoshi-yagi","given":"Satoshi","family":"Yagi","affiliation":"NTT Software Innovation Center"}],"abstract":"We propose a robust multivariate density estimator based on the variational autoencoder (VAE). The VAE is a powerful deep generative model, and used for multivariate density estimation. With the original VAE, the distribution of observed continuous variables is assumed to be a Gaussian, where its mean and variance are modeled by deep neural networks taking latent variables as their inputs. This distribution is called the decoder. However, the training of VAE often becomes unstable. One reason is that the decoder of VAE is sensitive to the error between the data point and its estimated mean when its estimated variance is almost zero. We solve this instability problem by making the decoder robust to the error using a Bayesian approach to the variance estimation: we set a prior for the variance of the Gaussian decoder, and marginalize it out analytically, which leads to proposing the Student-t VAE. Numerical experiments with various datasets show that training of the Student-t VAE is robust, and the Student-t VAE achieves high density estimation performance.","doi":"10.24963/ijcai.2018/374","url":"https://ijcai.org/proceedings/2018/374","pdf":"https://ijcai.org/proceedings/2018/0374.pdf","markdown":"https://ijcai.org/proceedings/2018/374.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/374","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":374,"pages":{"first":2696,"last":2702},"keywords":["Machine Learning: Neural Networks","Machine Learning: Machine Learning","Machine Learning: Deep Learning","Machine Learning: Probabilistic Machine Learning","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Burda et al., 2015] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519, 2015.","title":"Importance weighted autoencoders","authors":"Yuri Burda, Roger Grosse, Ruslan Salakhutdinov","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Geusebroek et al., 2005] Jan-Mark Geusebroek, Gertjan J Burghouts, and Arnold WM Smeulders. The amsterdam library of object images. International Journal of Com- puter Vision, 61(1):103-112, 2005.","title":"The amsterdam library of object images","authors":"Jan-Mark Geusebroek, Gertjan J Burghouts, Arnold Wm Smeulders","year":"2005","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goldstein and Uchida, 2016] Markus Goldstein and Seiichi Uchida. A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data. PloS one, 11(4):e0152173, 2016.","title":"A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data","authors":"Uchida Goldstein, Markus Goldstein, Seiichi Uchida","year":"2016","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":"http://www.deeplearningbook.org","internal_url":null},{"position":6,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information pro- cessing systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jylänki et al., 2011] Pasi Jylänki, Jarno Vanhatalo, and Aki Vehtari. Robust Gaussian process regression with a Student-t likelihood. Journal of Machine Learning Re- search, 12(Nov):3227-3257, 2011.","title":"Pasi Jylänki, Jarno Vanhatalo, and Aki Vehtari. Robust Gaussian process regression with a Student-t likelihood","authors":"Jylänki","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational Bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma et al., 2015] Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local repa- rameterization trick. In Advances in Neural Information Processing Systems, pages 2575-2583, 2015.","title":"Variational dropout and the local reparameterization trick","authors":"Tim Diederik P Kingma, Max Salimans, Welling","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lange et al., 1989] Kenneth L Lange, Roderick JA Little, and Jeremy MG Taylor. Robust statistical modeling us- ing the t distribution. Journal of the American Statistical Association, 84(408):881-896, 1989.","title":"Robust statistical modeling using the t distribution","authors":"Roderick Ja Kenneth L Lange, Jeremy Mg Little, Taylor","year":"1989","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lichman, 2013] M. Lichman. UCI machine learning repos- itory, 2013.","title":"UCI machine learning repository","authors":"M Lichman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Martinez-Cantin et al., 2017] Ruben Martinez-Cantin, Michael McCourt, and Kevin Tee. Robust Bayesian optimization with Student-t likelihood. arXiv preprint arXiv:1707.05729, 2017.","title":"Robust Bayesian optimization with Student-t likelihood","authors":"Ruben Martinez-Cantin, Michael Mccourt, Kevin Tee","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McLachlan and Peel, 2004] Geoffrey McLachlan and David Peel. Finite mixture models. John Wiley & Sons, 2004.","title":"Finite mixture models","authors":"Geoffrey Mclachlan, David Peel","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Miller et al., 2017] Andrew Miller, Nick Foti, Alexander D'Amour, and Ryan P Adams. Reducing reparameteriza- tion gradient variance. In Advances in Neural Information Processing Systems 30, pages 3711-3721, 2017.","title":"Reducing reparameterization gradient variance","authors":"Miller","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Re- search, 12(Oct):2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rezende et al., 2014] Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In Pro- ceedings of the 31st International Conference on Machine Learning, pages 1278-1286, 2014.","title":"Stochastic backpropagation and approximate inference in deep generative models","authors":"Shakir Danilo J Rezende, Daan Mohamed, Wierstra","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Roeder et al., 2017] Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: Simple, lower-variance gradient estimators for variational infer- ence. In Advances in Neural Information Processing Sys- tems, pages 6928-6937, 2017.","title":"Sticking the landing: Simple, lower-variance gradient estimators for variational inference","authors":"Geoffrey Roeder, Yuhuai Wu, David K Duvenaud","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Scott, 2015] David W Scott. Multivariate density estima- tion: theory, practice, and visualization. John Wiley & Sons, 2015.","title":"Multivariate density estimation: theory, practice, and visualization","authors":"David W Scott","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silverman, 1986] Bernard W Silverman. Density estimation for statistics and data analysis, volume 26. CRC press, 1986.","title":"Density estimation for statistics and data analysis","authors":"Bernard W Silverman","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Suh et al., 2016] Suwon Suh, Daniel H Chae, Hyon-Goo Kang, and Seungjin Choi. Echo-state conditional varia- tional autoencoder for anomaly detection. In Neural Net- works (IJCNN), 2016 International Joint Conference on, pages 1015-1022. IEEE, 2016.","title":"Echo-state conditional variational autoencoder for anomaly detection","authors":"Suh","year":"2016","venue":"Neural Networks (IJCNN), 2016 International Joint Conference on","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.","title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"COURSERA: Neural networks for machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[van den Oord et al., 2016] Aaron van den Oord, Nal Kalch- brenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with PixelCNN decoders. In Advances in Neural Information Processing Systems, pages 4790-4798, 2016.","title":"Conditional image generation with PixelCNN decoders","authors":"Oord Van Den","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2013] Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. In Advances in Neural Information Processing Systems, pages 181-189, 2013.","title":"Variance reduction for stochastic gradient optimization","authors":"Chong Wang, Xi Chen, Alexander J Smola, Eric P Xing","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeiler, 2012] Matthew D Zeiler. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.","title":"ADADELTA: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}