{"type":"ScholarlyArticle","title":"Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering","authors":[{"name":"Zhuxi Jiang","slug":"zhuxi-jiang","given":"Zhuxi","family":"Jiang","affiliation":"Beijing Institute of Technology , Beijing , China"},{"name":"Yin Zheng","slug":"yin-zheng","given":"Yin","family":"Zheng","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Huachun Tan","slug":"huachun-tan","given":"Huachun","family":"Tan","affiliation":"Beijing Institute of Technology , Beijing , China"},{"name":"Bangsheng Tang","slug":"bangsheng-tang","given":"Bangsheng","family":"Tang","affiliation":"Hulu LLC. , Beijing , China"},{"name":"Hanning Zhou","slug":"hanning-zhou","given":"Hanning","family":"Zhou","affiliation":"Hulu LLC. , Beijing , China"}],"abstract":"Clustering is among the most fundamental tasks in machine learning and artificial intelligence. In this paper, we propose Variational Deep Embedding (VaDE), a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE). Specifically, VaDE models the data generative procedure with a Gaussian Mixture Model (GMM) and a deep neural network (DNN): 1) the GMM picks a cluster; 2) from which a latent embedding is generated; 3) then the DNN decodes the latent embedding into an observable. Inference in VaDE is done in a variational way: a different DNN is used to encode observables to latent embeddings, so that the evidence lower bound (ELBO) can be optimized using the Stochastic Gradient Variational Bayes (SGVB) estimator and the reparameterization trick. Quantitative comparisons with strong baselines are included in this paper, and experimental results show that VaDE significantly outperforms the state-of-the-art clustering methods on 5 benchmarks from various modalities. Moreover, by VaDE's generative nature, we show its capability of generating highly realistic samples for any specified cluster, without using supervised information during training.","doi":"10.24963/ijcai.2017/273","url":"https://ijcai.org/proceedings/2017/273","pdf":"https://ijcai.org/proceedings/2017/0273.pdf","markdown":"https://ijcai.org/proceedings/2017/273.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/273","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":273,"pages":{"first":1965,"last":1972},"keywords":["Machine Learning: Machine Learning","Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbasnejad et al., 2016] Ehsan Abbasnejad, Anthony Dick, and Anton van den Hengel. Infinite variational au- toencoder for semi-supervised learning. arXiv preprint arXiv:1611.07800, 2016.","title":"Infinite variational autoencoder for semi-supervised learning","authors":"Ehsan Abbasnejad, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Coates et al., 2011] Adam Coates, Andrew Y Ng, and Honglak Lee. An analysis of single-layer networks in un- supervised feature learning. In International Conference on Artificial Intelligence and Statistics, 2011.","title":"An analysis of single-layer networks in unsupervised feature learning","authors":"Adam Coates, Andrew Y Ng, Honglak Lee","year":"2011","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dosovitskiy and Brox, 2016] Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In NIPS, 2016.","title":"Generating images with perceptual similarity metrics based on deep networks","authors":"Brox Dosovitskiy, Alexey Dosovitskiy, Thomas Brox","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Salimans, 2016] Diederik P Kingma and Tim Salimans. Improving variational autoencoders with in- verse autoregressive flow. In NIPS, 2016.","title":"Improving variational autoencoders with inverse autoregressive flow","authors":"P Diederik, Tim Kingma, Salimans","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma et al., 2014] Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi- supervised learning with deep generative models. In NIPS, 2014.","title":"Semisupervised learning with deep generative models","authors":"P Diederik, Danilo J Kingma, Shakir Rezende, Max Mohamed, Welling","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lewis et al., 2004] David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"Yiming David D Lewis, Tony G Yang, Fan Rose, Li","year":"2004","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2010] Jialu Liu, Deng Cai, and Xiaofei He. Gaus- sian mixture model with local consistency. In AAAI, 2010.","title":"Gaussian mixture model with local consistency","authors":"Jialu Liu, Deng Cai, Xiaofei He","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Maaløe et al., 2016] Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. In ICML, 2016.","title":"Auxiliary deep generative models","authors":"Maaløe","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.","title":"Visualizing data using t-sne","authors":"Hinton ; Maaten Maaten, Geoffrey Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Makhzani et al., 2016] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Ad- versarial autoencoders. In NIPS, 2016.","title":"Adversarial autoencoders","authors":"Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Munkres, 1957] James Munkres. Algorithms for the assign- ment and transportation problems. Journal of the society for industrial and applied mathematics, 1957.","title":"Algorithms for the assignment and transportation problems","authors":"James Munkres","year":"1957","venue":"Journal of the society for industrial and applied mathematics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nalisnick and Smyth, 2016] Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders. arXiv preprint arXiv:1605.06197, 2016.","title":"Stick-breaking variational autoencoders","authors":"Smyth Nalisnick","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nalisnick et al., 2016] Eric Nalisnick, Lars Hertel, and Padhraic Smyth. Approximate inference for deep latent gaussian mixtures. 2016.","title":"Approximate inference for deep latent gaussian mixtures","authors":"Eric Nalisnick, Lars Hertel, Padhraic Smyth","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ng et al., 2002] Andrew Y Ng, Michael I Jordan, Yair Weiss, et al. On spectral clustering: Analysis and an al- gorithm. In NIPS, 2002.","title":"On spectral clustering: Analysis and an algorithm","authors":"Y Andrew, Michael I Ng, Yair Jordan, Weiss","year":"2002","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen et al., 2016] Anh Nguyen, Jason Yosinski, Yoshua Bengio, Alexey Dosovitskiy, and Jeff Clune. Plug & play generative networks: Conditional iterative generation of images in latent space. arXiv preprint arXiv:1612.00005, 2016.","title":"Plug & play generative networks: Conditional iterative generation of images in latent space","authors":"Anh Nguyen, Jason Yosinski, Yoshua Bengio, Alexey Dosovitskiy, Jeff Clune","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Oord et al., 2016] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016.","title":"Pixel recurrent neural networks","authors":"Aaron Van Den Oord, Nal Kalchbrenner, Koray Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Radford et al., 2016] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Woj- ciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shu et al., 2016] Rui Shu, James Brofos, Frank Zhang, Hung Hai Bui, Mohammad Ghavamzadeh, and Mykel Kochenderfer. Stochastic video prediction with condi- tional density estimation. In ECCV Workshop on Action and Anticipation for Visual Learning, 2016.","title":"Stochastic video prediction with conditional density estimation","authors":"Rui Shu, James Brofos, Frank Zhang, Hung Hai Bui, Mohammad Ghavamzadeh, Mykel Kochenderfer","year":"2016","venue":"ECCV Workshop on Action and Anticipation for Visual Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sønderby et al., 2016] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NIPS, 2016.","title":"Ladder variational autoencoders","authors":"Tapani Casper Kaae Sønderby, Lars Raiko, Søren Maaløe, Ole Kaae Sønderby, Winther","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stisen et al., 2015] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjær- gaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmo- bile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, 2015.","title":"Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition","authors":"Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, Mads Møller Jensen","year":"2015","venue":"Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Von Luxburg, 2007] Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 2007.","title":"A tutorial on spectral clustering","authors":"Ulrike Von, Luxburg","year":"2007","venue":"Statistics and computing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xie et al., 2016] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.","title":"Unsupervised deep embedding for clustering analysis","authors":"Junyuan Xie, Ross Girshick, Ali Farhadi","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2010] Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang. Image clustering us- ing local discriminant models and global integration. IEEE Transactions on Image Processing, 2010.","title":"Image clustering using local discriminant models and global integration","authors":"Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, Yueting Zhuang","year":"2010","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ye et al., 2008] Jieping Ye, Zheng Zhao, and Mingrui Wu. Discriminative k-means for clustering. In NIPS, 2008.","title":"Discriminative k-means for clustering","authors":"Ye","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2014a] Yin Zheng, Richard S Zemel, Yu-Jin Zhang, and Hugo Larochelle. A neural autoregressive ap- proach to attention-based recognition. International Jour- nal of Computer Vision, 113(1):67-79, 2014.","title":"A neural autoregressive approach to attention-based recognition","authors":"Zheng","year":"2014","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zheng et al., 2014b] Yin Zheng, Yu-Jin Zhang, and H. Larochelle. Topic modeling of multimodal data: An autoregressive approach. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1370-1377, June 2014.","title":"Topic modeling of multimodal data: An autoregressive approach","authors":"Zheng","year":"2014","venue":"Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zheng et al., 2015] Y. Zheng, Yu-Jin Zhang, and H. Larochelle. A deep and autoregressive approach for topic modeling of multimodal data. Pattern Anal- ysis and Machine Intelligence, IEEE Transactions on, PP(99):1-1, 2015.","title":"A deep and autoregressive approach for topic modeling of multimodal data. Pattern Analysis and Machine Intelligence","authors":"Y Zheng, Yu-Jin Zhang, H Larochelle","year":"2015","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zheng et al., 2016] Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hanning Zhou. A neural autoregressive ap- proach to collaborative filtering. In Proceedings of the 33nd International Conference on Machine Learning, pages 764-773, 2016.","title":"A neural autoregressive approach to collaborative filtering","authors":"Yin Zheng, Bangsheng Tang, Wenkui Ding, Hanning Zhou","year":"2016","venue":"Proceedings of the 33nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}