{"type":"ScholarlyArticle","title":"Improved Deep Embedded Clustering with Local Structure Preservation","authors":[{"name":"Xifeng Guo","slug":"xifeng-guo","given":"Xifeng","family":"Guo","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Long Gao","slug":"long-gao","given":"Long","family":"Gao","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Xinwang Liu","slug":"xinwang-liu","given":"Xinwang","family":"Liu","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Jianping Yin","slug":"jianping-yin","given":"Jianping","family":"Yin","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"}],"abstract":"Deep clustering learns deep feature representations that favor clustering task using neural networks. Some pioneering work proposes to simultaneously learn embedded features and perform clustering by explicitly defining a clustering oriented loss. Though promising performance has been demonstrated in various applications, we observe that a vital ingredient has been overlooked by these work that the defined clustering loss may corrupt feature space, which leads to non-representative meaningless features and this in turn hurts clustering performance. To address this issue, in this paper, we propose the Improved Deep Embedded Clustering (IDEC) algorithm to take care of data structure preservation. Specifically, we manipulate feature space to scatter data points using a clustering loss as guidance. To constrain the manipulation and maintain the local structure of data generating distribution, an under-complete autoencoder is applied. By integrating the clustering loss and autoencoder's reconstruction loss, IDEC can jointly optimize cluster labels assignment and learn features that are suitable for clustering with local structure preservation. The resultant optimization problem can be effectively solved by mini-batch stochastic gradient descent and backpropagation. Experiments on image and text datasets empirically validate the importance of local structure preservation and the effectiveness of our algorithm.","doi":"10.24963/ijcai.2017/243","url":"https://ijcai.org/proceedings/2017/243","pdf":"https://ijcai.org/proceedings/2017/0243.pdf","markdown":"https://ijcai.org/proceedings/2017/243.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/243","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":243,"pages":{"first":1753,"last":1759},"keywords":["Machine Learning: Deep Learning","Machine Learning: Machine Learning","Machine Learning: Unsupervised Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006.","title":"Pattern Recognition and Machine Learning","authors":"M Christopher, Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen, 2015] Gang Chen. Deep learning with nonparametric clustering. arXiv preprint arXiv:1501.03084, 2015.","title":"Deep learning with nonparametric clustering","authors":"Gang Chen","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chollet, 2015] Franc ¸ois Chollet. Keras, 2015.","title":null,"authors":"Chollet Franc, Keras","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural network- s. Journal of Machine Learning Research, 15:315-323, 2011.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT Press, 2016.","title":"Deep learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lewis et al., 2004] David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5(Apr):361-397, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"Yiming David D Lewis, Tony G Yang, Fan Rose, Li","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[MacQueen, 1967] James MacQueen. Some methods for classification and analysis of multivariate observations. In Berkeley Symposium on Mathematical Statistics and Prob- ability, volume 1, pages 281-297.","title":"Some methods for classification and analysis of multivariate observations","authors":"James Macqueen","year":null,"venue":"Berkeley Symposium on Mathematical Statistics and Probability","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Oakland, CA, USA., 1967.","title":null,"authors":"C A Oakland, Usa","year":"1967","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nie et al., 2011] Feiping Nie, Zinan Zeng, Ivor W Tsang, Dong Xu, and Changshui Zhang. Spectral embedded clustering: A framework for in-sample and out-of-sample spectral clustering. IEEE Transactions on Neural Net- works, 22(11):1796-1808, 2011.","title":"Spectral embedded clustering: A framework for in-sample and out-of-sample spectral clustering","authors":"Nie","year":"2011","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nigam and Ghani, 2000] Kamal Nigam and Rayid Ghani. Analyzing the effectiveness and applicability of co- training. In International Conference on Information and Knowledge Management, pages 86-93. ACM, 2000.","title":"Analyzing the effectiveness and applicability of cotraining","authors":"Ghani Nigam, Kamal Nigam, Rayid Ghani","year":"2000","venue":"International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Peng et al., 2016] Xi Peng, Shijie Xiao, Jiashi Feng, Wei- Yun Yau, and Zhang Yi. Deep subspace clustering with sparsity prior. In International Joint Conference on Artifi- cial Intelligence (IJCAI), 2016.","title":"Deep subspace clustering with sparsity prior","authors":"Xi Peng, Shijie Xiao, Jiashi Feng, Wei-Yun Yau, Zhang Yi","year":"2016","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2016/275"},{"position":15,"raw":"[Tian et al., 2014] Fei Tian, Bin Gao, Qing Cui, Enhong Chen, and Tie-Yan Liu. Learning deep representations for graph clustering. In AAAI, pages 1293-1299, 2014.","title":"Learning deep representations for graph clustering","authors":"Bin Fei Tian, Qing Gao, Enhong Cui, Tie-Yan Chen, Liu","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Von Luxburg, 2007] Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17(4):395- 416, 2007.","title":"A tutorial on spectral clustering","authors":"Ulrike Von, Luxburg","year":"2007","venue":"Statistics and Computing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xie et al., 2016] Junyuan Xie, Ross Girshick, and Al- i Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learn- ing (ICML), 2016.","title":"Unsupervised deep embedding for clustering analysis","authors":"Junyuan Xie, Ross Girshick, Ali Farhadi","year":"2016","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2016] Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 5147-5156, 2016.","title":"Joint unsupervised learning of deep representations and image clusters","authors":"Jianwei Yang, Devi Parikh, Dhruv Batra","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zheng et al., 2016] Yin Zheng, Huachun Tan, Bangsheng Tang, Hanning Zhou, et al. Variational deep embedding: A generative approach to clustering. arXiv preprint arX- iv:1611.05148, 2016.","title":"Variational deep embedding: A generative approach to clustering","authors":"Yin Zheng, Huachun Tan, Bangsheng Tang, Hanning Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}