{"type":"ScholarlyArticle","title":"Improving representation learning in autoencoders via multidimensional interpolation and dual regularizations","authors":[{"name":"Sheng Qian","slug":"sheng-qian","given":"Sheng","family":"Qian","affiliation":"Huawei Device Company Limited"},{"name":"Guanyue Li","slug":"guanyue-li","given":"Guanyue","family":"Li","affiliation":"School of Computer Science and Engineering , South China University of Technology"},{"name":"Wen-Ming Cao","slug":"wen-ming-cao","given":"Wen-Ming","family":"Cao","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Cheng Liu","slug":"cheng-liu","given":"Cheng","family":"Liu","affiliation":"Department of Computer Science , Shantou University"},{"name":"Si Wu","slug":"si-wu","given":"Si","family":"Wu","affiliation":"School of Computer Science and Engineering , South China University of Technology"},{"name":"Hau San Wong","slug":"hau-san-wong","given":"Hau San","family":"Wong","affiliation":"Department of Computer Science , City University of Hong Kong"}],"abstract":"Autoencoders enjoy a remarkable ability to learn data representations. Research on autoencoders shows that the effectiveness of data interpolation can reflect the performance of representation learning. However, existing interpolation methods in autoencoders do not have enough capability of traversing a possible region between two datapoints on a data manifold, and the distribution of interpolated latent representations is not considered.To address these issues, we aim to fully exert the potential of data interpolation and further improve representation learning in autoencoders. Specifically, we propose the multidimensional interpolation to increase the capability of data interpolation by randomly setting interpolation coefficients for each dimension of latent representations. In addition, we regularize autoencoders in both the latent and the data spaces by imposing a prior on latent representations in the Maximum Mean Discrepancy (MMD) framework and encouraging generated datapoints to be realistic in the Generative Adversarial Network (GAN) framework. Compared to representative models, our proposed model has empirically shown that representation learning exhibits better performance on downstream tasks on multiple benchmarks.","doi":"10.24963/ijcai.2019/453","url":"https://ijcai.org/proceedings/2019/453","pdf":"https://ijcai.org/proceedings/2019/0453.pdf","markdown":"https://ijcai.org/proceedings/2019/453.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/453","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":453,"pages":{"first":3268,"last":3274},"keywords":["Machine Learning: Deep Learning","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Agustsson et al., 2019] Eirikur Agustsson, Alexander Sage, Radu Timofte, and Luc Van Gool. Optimal transport maps for distribution preserving operations on latent spaces of generative models. In International Conference on Learning Representations, 2019.","title":"Optimal transport maps for distribution preserving operations on latent spaces of generative models","authors":"Eirikur Agustsson, Alexander Sage, Radu Timofte, Luc Van Gool","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berthelot et al., 2019] David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. In International Conference on Learning Representations, 2019.","title":"Understanding and improving interpolation in autoencoders via an adversarial regularizer","authors":"David Berthelot, Colin Raffel, Aurko Roy, Ian Goodfellow","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Coates et al., 2011] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics, pages 215-223, 2011.","title":"An analysis of single-layer networks in unsupervised feature learning","authors":"Adam Coates, Andrew Ng, Honglak Lee","year":"2011","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dumoulin et al., 2017] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. In International Conference on Learning Representations, 2017.","title":"Adversarially learned inference","authors":"Ishmael Vincent Dumoulin, Ben Belghazi, Olivier Poole, Alex Mastropietro, Martin Lamb, Aaron Arjovsky, Courville","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, pages 2672- 2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gretton et al., 2007] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex J Smola. A kernel method for the two-sample-problem. In Neural Information Processing Systems, pages 513-520, 2007.","title":"A kernel method for the two-sample-problem","authors":"Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, Alex J Smola","year":"2007","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2017] Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International Conference on Machine Learning, pages 1558-1567, 2017.","title":"Learning discrete representations via information maximizing self-augmented training","authors":"Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, Masashi Sugiyama","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Laine, 2018] Samuli Laine. Feature-based metrics for exploring the latent space of generative models. In International Conference on Learning Representations, 2018.","title":"Feature-based metrics for exploring the latent space of generative models","authors":"Laine ;, Samuli Laine","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Larsen et al., 2016] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International Conference on Machine Learning, pages 1558-1566, 2016.","title":"Autoencoding beyond pixels using a learned similarity metric","authors":"Larsen","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278- 2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee Rodgers and Nicewander, 1988] Joseph Lee Rodgers and W Alan Nicewander. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59-66, 1988.","title":"Thirteen ways to look at the correlation coefficient","authors":"Joseph, Lee Rodgers, Alan Nicewander","year":"1988","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[MacQueen, 1967] James MacQueen. Some methods for classification and analysis of multivariate observations. In Berkeley Symposium on Mathematical Statistics and Probability, pages 281-297, 1967.","title":"Some methods for classification and analysis of multivariate observations","authors":"James Macqueen, Macqueen","year":"1967","venue":"Berkeley Symposium on Mathematical Statistics and Probability","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Makhzani et al., 2016] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. In International Conference on Learning Representations, 2016.","title":"Adversarial autoencoders","authors":"Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tolstikhin et al., 2018] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.","title":"Wasserstein auto-encoders","authors":"Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, Bernhard Schoelkopf","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van den Oord et al., 2017] Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Neural Information Processing Systems, pages 6306-6315, 2017.","title":"Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(12):3371-3408, 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[White, 2016] Tom White. Sampling generative networks. arXiv preprint arXiv:1609.04468, 2016.","title":"Sampling generative networks","authors":"Tom White","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2016] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Neural Information Processing Systems, pages 82-90, 2016.","title":"Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling","authors":"Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, Josh Tenenbaum","year":"2016","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2016] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pages 478-487, 2016.","title":"Unsupervised deep embedding for clustering analysis","authors":"Junyuan Xie, Ross Girshick, Ali Farhadi","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2016] Jun Xu, Lei Xiang, Qingshan Liu, Hannah Gilmore, Jianzhong Wu, Jinghai Tang, and Anant Madabhushi. Stacked sparse autoencoder (ssae) for nuclei detection on breast cancer histopathology images. IEEE Transactions on Medical Imaging, 35(1):119-130, 2016.","title":"Stacked sparse autoencoder (ssae) for nuclei detection on breast cancer histopathology images","authors":"Jun Xu, Lei Xiang, Qingshan Liu, Hannah Gilmore, Jianzhong Wu, Jinghai Tang, Anant Madabhushi","year":"2016","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}