{"type":"ScholarlyArticle","title":"Unsupervised Representation Learning by Predicting Random Distances","authors":[{"name":"Hu Wang","slug":"hu-wang","given":"Hu","family":"Wang","affiliation":"The University of Adelaide , Australia"},{"name":"Guansong Pang","slug":"guansong-pang","given":"Guansong","family":"Pang","affiliation":"The University of Adelaide , Australia"},{"name":"Chunhua Shen","slug":"chunhua-shen","given":"Chunhua","family":"Shen","affiliation":"The University of Adelaide , Australia"},{"name":"Congbo Ma","slug":"congbo-ma","given":"Congbo","family":"Ma","affiliation":"The University of Adelaide , Australia"}],"abstract":"Deep neural networks have gained great success in a broad range of tasks due to its remarkable capability to learn semantically rich features from high-dimensional data. However, they often require large-scale labelled data to successfully learn such features, which significantly hinders their adaption in unsupervised learning tasks, such as anomaly detection and clustering, and limits their applications to critical domains where obtaining massive labelled data is prohibitively expensive. To enable unsupervised learning on those domains, in this work we propose to learn features without using any labelled data by training neural networks to predict data distances in a randomly projected space. Random mapping is a theoretically  proven approach to obtain approximately preserved distances. To well predict these distances, the representation learner is optimised to learn genuine class structures that are implicitly embedded in the randomly projected space. Empirical results on 19 real-world datasets show that our learned representations substantially outperform a few state-of-the-art methods for both anomaly detection and clustering tasks. Code is available at: \\url{https://git.io/RDP}","doi":"10.24963/ijcai.2020/408","url":"https://ijcai.org/proceedings/2020/408","pdf":"https://ijcai.org/proceedings/2020/0408.pdf","markdown":"https://ijcai.org/proceedings/2020/408.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/408","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":408,"pages":{"first":2950,"last":2956},"keywords":["Data Mining: Feature Extraction, Selection and Dimensionality Reduction","Machine Learning: Unsupervised Learning","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24133/file_24133.m3u8","length_sec":290,"provider":"hls"}],"references":[{"position":1,"raw":"[Agrawal et al., 2015] Pulkit Agrawal, Joao Carreira, and Ji- tendra Malik. Learning to see by moving. In Proc. IEEE Int. Conf. Comp. Vis., pages 37-45, 2015.","title":"Learning to see by moving","authors":"Pulkit Agrawal, Joao Carreira, Jitendra Malik","year":"2015","venue":"Proc. IEEE Int. Conf. Comp. Vis","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beyer et al., 1999] Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is \"nearest neighbor\" meaningful? In Proc. Int. Conf. Database Theory, 1999.","title":"When is \"nearest neighbor\" meaningful?","authors":"Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, Uri Shaft","year":"1999","venue":"Proc. Int. Conf. Database Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bingham and Mannila, 2001] Ella Bingham and Heikki Mannila. Random projection in dimensionality reduc- tion: applications to image and text data. In Proc. ACM SIGKDD Int. Conf. Know. Disco. & Data Mining, 2001.","title":"Random projection in dimensionality reduction: applications to image and text data","authors":"Mannila Bingham, Ella Bingham, Heikki Mannila","year":"2001","venue":"Proc. ACM SIGKDD Int. Conf. Know. Disco. & Data Mining","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burda et al., 2019] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random net- work distillation. In Proc. Int. Conf. Learn. Repre., 2019.","title":"Exploration by random network distillation","authors":"Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov","year":"2019","venue":"Proc. Int. Conf. Learn. Repre","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In Proc. Adv. Neu- ral Inf. Process. Syst., pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"Proc. Adv. Neural Inf. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Demšar, 2006] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 2006.","title":"Statistical comparisons of classifiers over multiple data sets","authors":"Janez Demšar","year":"2006","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Doersch et al., 2015] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learn- ing by context prediction. In Proc. IEEE Int. Conf. Comp. Vis., pages 1422-1430, 2015.","title":"Unsupervised visual representation learning by context prediction","authors":"Carl Doersch, Abhinav Gupta, Alexei A Efros","year":"2015","venue":"Proc. IEEE Int. Conf. Comp. Vis","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Donoho and Grimes, 2003] David L Donoho and Carrie Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10):5591-5596, 2003.","title":"Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data","authors":"L David, Carrie Donoho, Grimes","year":"2003","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Proc. Int. Conf. Learn. Repre., 2018.","title":"Unsupervised representation learning by predicting image rotations","authors":"Spyros Gidaris, Praveer Singh, Nikos Komodakis","year":"2018","venue":"Proc. Int. Conf. Learn. Repre","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hartigan and Wong, 1979] John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. J. Royal Stat. Society, 1979.","title":"Algorithm as 136: A k-means clustering algorithm","authors":"A John, Manchek A Hartigan, Wong","year":"1979","venue":"J. Royal Stat. Society","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hinton and Roweis, 2003] Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. In Proc. Adv. Neural Inf. Process. Syst., pages 857-864, 2003.","title":"Stochastic neighbor embedding","authors":"Geoffrey E Hinton, Sam T Roweis","year":"2003","venue":"Proc. Adv. Neural Inf. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hinton and Salakhutdinov, 2006] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.","title":"Reducing the dimensionality of data with neural networks","authors":"Geoffrey E Hinton, Ruslan R Salakhutdinov","year":"2006","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson and Lindenstrauss, 1984] William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary Math., 1984.","title":"Extensions of lipschitz mappings into a hilbert space","authors":"B William, Joram Johnson, Lindenstrauss","year":"1984","venue":"Contemporary Math","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In Proc. Int. Conf. Mach. Learn., pages 1188-1196, 2014.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":"Proc. Int. Conf. Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2017] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. In Proc. IEEE Int. Conf. Comp. Vis., pages 667-676, 2017.","title":"Unsupervised representation learning by sorting sequences","authors":"Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, Ming-Hsuan Yang","year":"2017","venue":"Proc. IEEE Int. Conf. Comp. Vis","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2006] Ping Li, Trevor J Hastie, and Kenneth W Church. Very sparse random projections. In Proc. ACM SIGKDD Int. Conf. Know. Disco. & Data Mining, 2006.","title":"Very sparse random projections","authors":"Ping Li, Trevor J Hastie, Kenneth W Church","year":"2006","venue":"Proc. ACM SIGKDD Int. Conf. Know. Disco. & Data Mining","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2008] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In Proc. IEEE Int. Conf. Data Min- ing, pages 413-422. IEEE, 2008.","title":"Isolation forest","authors":"Tony Fei, Kai Liu, Ming Ting, Zhi-Hua Zhou","year":"2008","venue":"Proc. IEEE Int. Conf. Data Mining","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Misra et al., 2016] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learn- ing using temporal order verification. In Proc. Eur. Conf. Comp. Vis., pages 527-544. Springer, 2016.","title":"Shuffle and learn: unsupervised learning using temporal order verification","authors":"Ishan Misra, Lawrence Zitnick, Martial Hebert","year":"2016","venue":"Proc. Eur. Conf. Comp. Vis","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pang et al., 2018] Guansong Pang, Longbing Cao, Ling Chen, and Huan Liu. Learning representations of ultrahigh-dimensional data for random distance-based out- lier detection. In Proc. ACM SIGKDD Int. Conf. Know. Disco. & Data Mining, pages 2041-2050. ACM, 2018.","title":"Learning representations of ultrahigh-dimensional data for random distance-based outlier detection","authors":"Pang","year":"2018","venue":"Proc. ACM SIGKDD Int. Conf. Know. Disco. & Data Mining","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"Proc. IEEE Conf. Comp. Vis. Patt. Recogn","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proc. Int. Conf. Learn. Repre., 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":"Proc. Int. Conf. Learn. Repre","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rahimi and Recht, 2008] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Proc. Adv. Neural Inf. Process. Syst., 2008.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2008","venue":"Proc. Adv. Neural Inf. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rahmani and Atia, 2017] Mostafa Rahmani and George Atia. Coherence pursuit: Fast, simple, and robust sub- space recovery. In Proc. Int. Conf. Mach. Learn., pages 2864-2873. JMLR. org, 2017.","title":"Coherence pursuit: Fast, simple, and robust subspace recovery","authors":"Mostafa Rahmani, George Atia","year":"2017","venue":"Proc. Int. Conf. Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schölkopf et al., 1997] Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal com- ponent analysis. In Proc. Int. Conf. Arti. Neur. Net., pages 583-588. Springer, 1997.","title":"Kernel principal component analysis","authors":"Bernhard Schölkopf, Alexander Smola, Klaus-Robert Müller","year":"1997","venue":"Proc. Int. Conf. Arti. Neur. Net","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vempala, 1998] Santosh Vempala. Random projection: A new approach to VLSI layout. In Proc. Ann. Symp. Found. Comp Sci., pages 389-395. IEEE, 1998.","title":"Random projection: A new approach to VLSI layout","authors":"Santosh Vempala","year":"1998","venue":"Proc. Ann. Symp. Found. Comp Sci","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is- abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man- zagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res., 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2017] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 1851-1858, 2017.","title":"Unsupervised learning of depth and ego-motion from video","authors":"Zhou","year":"2017","venue":"Proc. IEEE Conf. Comp. Vis. Patt. Recogn","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zong et al., 2018] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Proc. Int. Conf. Learn. Repre., 2018.","title":"Deep autoencoding gaussian mixture model for unsupervised anomaly detection","authors":"Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, Haifeng Chen","year":"2018","venue":"Proc. Int. Conf. Learn. Repre","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}