{"type":"ScholarlyArticle","title":"Stacked Similarity-Aware Autoencoders","authors":[{"name":"Wenqing Chu","slug":"wenqing-chu","given":"Wenqing","family":"Chu","affiliation":"State Key Lab of CAD&CG , College of Computer Science , Zhejiang University , China"},{"name":"Deng Cai","slug":"deng-cai","given":"Deng","family":"Cai","affiliation":"State Key Lab of CAD&CG , College of Computer Science , Zhejiang University , China"}],"abstract":"As one of the most popular unsupervised learning approaches, the autoencoder aims at transforming the inputs to the outputs with the least discrepancy. The conventional autoencoder and most of its variants only consider the one-to-one reconstruction, which ignores the intrinsic structure of the data and may lead to overfitting. In order to preserve the latent geometric information in the data, we propose the stacked similarity-aware autoencoders. To train each single autoencoder, we first obtain the pseudo class label of each sample by clustering the input features. Then the hidden codes of those samples sharing the same category label will be required to satisfy an additional similarity constraint. Specifically, the similarity constraint is implemented based on an extension of the recently proposed center loss. With this joint supervision of the autoencoder reconstruction error and the center loss, the learned feature representations not only can reconstruct the original data, but also preserve the geometric structure of the data. Furthermore, a stacked framework is introduced to boost the representation capacity. The experimental results on several benchmark datasets show the remarkable performance improvement of the proposed algorithm compared with other autoencoder based approaches.","doi":"10.24963/ijcai.2017/216","url":"https://ijcai.org/proceedings/2017/216","pdf":"https://ijcai.org/proceedings/2017/0216.pdf","markdown":"https://ijcai.org/proceedings/2017/216.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/216","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":216,"pages":{"first":1561,"last":1567},"keywords":["Machine Learning: Neural Networks","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Boureau et al., 2008] Y-lan Boureau, Yann L Cun, et al. Sparse feature learning for deep belief networks. In NIPS, 2008.","title":"Sparse feature learning for deep belief networks","authors":"Y-Lan Boureau, Yann L Cun","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2011] Deng Cai, Hujun Bao, and Xiaofei He. Sparse concept coding for visual analysis. In CVPR, 2011.","title":"Sparse concept coding for visual analysis","authors":"Deng Cai, Hujun Bao, Xiaofei He","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ding et al., 2016] Zhengming Ding, Ming Shao, and Yun Fu. Deep robust encoder through locality preserving low- rank dictionary. In ECCV, 2016.","title":"Deep robust encoder through locality preserving lowrank dictionary","authors":"Zhengming Ding, Ming Shao, Yun Fu","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":"http://www.deeplearningbook.org","internal_url":null},{"position":5,"raw":"[Gowda and Krishna, 1978] K. Chidananda Gowda and G. Krishna. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern Recognition, 10:105-112, 1978.","title":"Agglomerative clustering using the concept of mutual nearest neighbourhood","authors":"K, Chidananda Gowda, G Krishna","year":"1978","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2005] Xiaofei He, Deng Cai, Shuicheng Yan, and HongJiang Zhang. Neighborhood preserving embedding. In ICCV, 2005.","title":"Neighborhood preserving embedding","authors":"Xiaofei He, Deng Cai, Shuicheng Yan, Hongjiang Zhang","year":"2005","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton and Salakhutdinov, 2006] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.","title":"Reducing the dimensionality of data with neural networks","authors":"Geoffrey E Hinton, Ruslan R Salakhutdinov","year":"2006","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jain et al., 1999] Anil K. Jain, M. Narasimha Murty, and Patrick J. Flynn. Data clustering: A review. ACM Comput. Surv., 31:264-323, 1999.","title":"Data clustering: A review","authors":"K Anil, M Narasimha Jain, Patrick J Murty, Flynn","year":"1999","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma et al., 2014] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In NIPS, 2014.","title":"Semisupervised learning with deep generative models","authors":"Shakir Diederik P Kingma, Danilo Mohamed, Max Jimenez Rezende, Welling","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lecun et al., 1998] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee, 2013] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, 2013.","title":"Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks","authors":"Dong-Hyun Lee","year":"2013","venue":"Workshop on Challenges in Representation Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Makhzani and Frey, 2014] Alireza Makhzani and Brendan Frey. k-sparse autoencoders. In ICLR, 2014.","title":"k-sparse autoencoders","authors":"Alireza Makhzani, Brendan Frey","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Makhzani and Frey, 2015] Alireza Makhzani and Brendan J Frey. Winner-take-all autoencoders. In NIPS, 2015.","title":"Winner-take-all autoencoders","authors":"Alireza Makhzani, Brendan J Frey","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Patel et al., 2016] Ankit B Patel, Minh Tan Nguyen, and Richard Baraniuk. A probabilistic framework for deep learning. In NIPS, 2016.","title":"A probabilistic framework for deep learning","authors":"B Ankit, Minh Tan Patel, Richard Nguyen, Baraniuk","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Plchot et al., 2016] Oldrich Plchot, Lukas Burget, Hagai Aronowitz, Pavel Mat, et al. Audio enhancing with dnn autoencoder for speaker recognition. In ICASSP, 2016.","title":"Audio enhancing with dnn autoencoder for speaker recognition","authors":"Oldrich Plchot, Lukas Burget, Hagai Aronowitz, Pavel Mat","year":"2016","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qian et al., 2016] Wei Qian, Bin Hong, Deng Cai, Xiaofei He, and Xuelong Li. Non-negative matrix factorization with sinkhorn distance. In IJCAI, 2016.","title":"Non-negative matrix factorization with sinkhorn distance","authors":"Wei Qian, Bin Hong, Deng Cai, Xiaofei He, Xuelong Li","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/280"},{"position":19,"raw":"[Rasmus et al., 2015] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In NIPS, 2015.","title":"Semisupervised learning with ladder networks","authors":"Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, Tapani Raiko","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rifai et al., 2011a] Salah Rifai, Yann Dauphin, Pascal Vin- cent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. In NIPS, 2011.","title":"The manifold tangent classifier","authors":"Rifai","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rifai et al., 2011b] Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extrac- tion. In ICML, 2011.","title":"Contractive auto-encoders: Explicit invariance during feature extraction","authors":"Rifai","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rumelhart et al., 1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Cognitive modeling, 5(3):1, 1988.","title":"Learning representations by back-propagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1988","venue":"Cognitive modeling","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schlkopf et al., 1998] Bernhard Schlkopf, Alexander Smola, and Klaus Robert Mller. Nonlinear compo- nent analysis as a kernel eigenvalue problem. Neural Computation, 10(5):1299-1319, 1998.","title":"Nonlinear component analysis as a kernel eigenvalue problem","authors":"Bernhard Schlkopf, Alexander Smola, Klaus Robert Mller","year":"1998","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Turk and Pentland, 1991] M Turk and A Pentland. Eigen- faces for recognition. Journal of cognitive neuroscience, 3 1:71-86, 1991.","title":"Eigenfaces for recognition","authors":"M Turk, Pentland","year":"1991","venue":"Journal of cognitive neuroscience","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is- abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man- zagol. Stacked denoising autoencoders: Learning use- ful representations in a deep network with a local de- noising criterion. Journal of Machine Learning Research, 11(Dec):3371-3408, 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2015] Hao Wang, Xingjian Shi, and Dit-Yan Yeung. Relational stacked denoising autoencoder for tag recommendation. In AAAI, 2015.","title":"Relational stacked denoising autoencoder for tag recommendation","authors":"Hao Wang, Xingjian Shi, Dit-Yan Yeung","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2016] Shuyang Wang, Zhengming Ding, and Yun Fu. Coupled marginalized auto-encoders for cross- domain multi-view learning. In IJCAI, 2016.","title":"Coupled marginalized auto-encoders for crossdomain multi-view learning","authors":"Shuyang Wang, Zhengming Ding, Yun Fu","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wen et al., 2016] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning ap- proach for deep face recognition. In ECCV, 2016.","title":"A discriminative feature learning approach for deep face recognition","authors":"Yandong Wen, Kaipeng Zhang, Zhifeng Li, Yu Qiao","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2016] Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016.","title":"Joint unsupervised learning of deep representations and image clusters","authors":"Jianwei Yang, Devi Parikh, Dhruv Batra","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2013] Wenchao Yu, Guangxiang Zeng, Ping Luo, Fuzhen Zhuang, Qing He, and Zhongzhi Shi. Embedding with autoencoder regularization. In Joint European Con- ference on Machine Learning and Knowledge Discovery in Databases, 2013.","title":"Embedding with autoencoder regularization","authors":"Wenchao Yu, Guangxiang Zeng, Ping Luo, Fuzhen Zhuang, Qing He, Zhongzhi Shi","year":"2013","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2015] Junbo Zhao, Michael Mathieu, Ross Goroshin, and Yann Lecun. Stacked what-where auto- encoders. arXiv preprint arXiv:1506.02351, 2015.","title":"Stacked what-where autoencoders","authors":"Junbo Zhao, Michael Mathieu, Ross Goroshin, Yann Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2016] Zhou Zhao, Xiaofei He, Deng Cai, Li- jun Zhang, Wilfred Ng, and Yueting Zhuang. Graph regularized feature selection with data reconstruction. IEEE Transactions on Knowledge and Data Engineering, 28(3):689-700, 2016.","title":"Graph regularized feature selection with data reconstruction","authors":"Zhou Zhao, Xiaofei He, Deng Cai, Lijun Zhang, Wilfred Ng, Yueting Zhuang","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}