{"type":"ScholarlyArticle","title":"ActUp: Analyzing and Consolidating tSNE and UMAP","authors":[{"name":"Andrew Draganov","slug":"andrew-draganov","given":"Andrew","family":"Draganov","affiliation":"Aarhus University"},{"name":"Jakob Jørgensen","slug":"jakob-j-rgensen","given":"Jakob","family":"Jørgensen","affiliation":"Aarhus University"},{"name":"Katrine Scheel","slug":"katrine-scheel","given":"Katrine","family":"Scheel","affiliation":"Aarhus University"},{"name":"Davide Mottin","slug":"davide-mottin","given":"Davide","family":"Mottin","affiliation":"Aarhus University"},{"name":"Ira Assent","slug":"ira-assent","given":"Ira","family":"Assent","affiliation":"Aarhus University"},{"name":"Tyrus Berry","slug":"tyrus-berry","given":"Tyrus","family":"Berry","affiliation":"George Mason University"},{"name":"Cigdem Aslay","slug":"cigdem-aslay","given":"Cigdem","family":"Aslay","affiliation":"Aarhus University"}],"abstract":"TSNE and UMAP are popular dimensionality reduction algorithms due to their speed and interpretable low-dimensional embeddings. Despite their popularity, however, little work has been done to study their full span of differences. We theoretically and experimentally evaluate the space of parameters in the TSNE and UMAP algorithms and observe that a single one -- the normalization -- is responsible for switching between them. This, in turn, implies that a majority of the algorithmic differences can be toggled without affecting the embeddings. We discuss the implications this has on several theoretic claims behind UMAP, as well as how to reconcile them with existing TSNE interpretations.\r\n\r\nBased on our analysis, we provide a method (GDR) that combines previously incompatible techniques from TSNE and UMAP and can replicate the results of either algorithm. This allows our method to incorporate further improvements, such as an acceleration that obtains either method's outputs faster than UMAP. We release improved versions of TSNE, UMAP, and GDR that are fully plug-and-play with the traditional libraries.","doi":"10.24963/ijcai.2023/406","url":"https://ijcai.org/proceedings/2023/406","pdf":"https://ijcai.org/proceedings/2023/0406.pdf","markdown":"https://ijcai.org/proceedings/2023/406.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/406","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":406,"pages":{"first":3651,"last":3658},"keywords":["Data Mining: DM: Data visualization","Machine Learning: ML: Unsupervised learning","Machine Learning: ML: Feature extraction, selection and dimensionality reduction"],"videos":[],"references":[{"position":1,"raw":"[Belkin and Niyogi, 2003] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality re- duction and data representation. Neural computation, 15(6):1373-1396, 2003.","title":"Laplacian eigenmaps for dimensionality reduction and data representation","authors":"Niyogi Belkin, Mikhail Belkin, Partha Niyogi","year":"2003","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bohm et al., 2020] Jan Niklas Bohm, Philipp Berens, and Dmitry Kobak. A unifying perspective on neighbor em- beddings along the attraction-repulsion spectrum. arXiv preprint arXiv:2007.08902, 2020.","title":"A unifying perspective on neighbor embeddings along the attraction-repulsion spectrum","authors":"Bohm","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Damrich and Hamprecht, 2021] Sebastian Damrich and Fred A Hamprecht. On umap's true loss function. Advances in Neural Information Processing Systems, 34, 2021.","title":"Sebastian Damrich and Fred A Hamprecht. On umap's true loss function","authors":"Hamprecht Damrich","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Damrich et al., 2022] Sebastian Damrich, Jan Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak. Con- trastive learning unifies t-sne and umap. arXiv preprint arXiv:2206.01816, 2022.","title":"Contrastive learning unifies t-sne and umap","authors":"Sebastian Damrich, Jan Niklas Böhm, Fred A Hamprecht, Dmitry Kobak","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng, 2012] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.","title":"The mnist database of handwritten digit images for machine learning research","authors":"Li Deng","year":"2012","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2011] Wei Dong, Charikar Moses, and Kai Li. Efficient k-nearest neighbor graph construction for generic similarity measures. In Proceedings of the 20th inter- national conference on World wide web, pages 577-586, 2011.","title":"Efficient k-nearest neighbor graph construction for generic similarity measures","authors":"Wei Dong, Charikar Moses, Kai Li","year":"2011","venue":"Proceedings of the 20th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hull, 1994] J. J. Hull. A database for handwritten text recognition research. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 16(5):550-554, 1994.","title":"A database for handwritten text recognition research","authors":"J J Hull","year":"1994","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kobak and Linderman, 2021] Dmitry Kobak and George C Linderman. Initialization is critical for preserving global data structure in both t-sne and umap. Nature biotechnol- ogy, 39(2):156-157, 2021.","title":"Initialization is critical for preserving global data structure in both t-sne and umap","authors":"Dmitry Kobak, George C Linderman","year":"2021","venue":"Nature biotechnology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical Report, Uni- versity of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Linderman et al., 2019] George C Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. Fast interpolation-based t-sne for improved visualization of single-cell rna-seq data. Nature methods, 16(3):243-245, 2019.","title":"Fast interpolation-based t-sne for improved visualization of single-cell rna-seq data","authors":"Manas George C Linderman, Jeremy G Rachh, Stefan Hoskins, Yuval Steinerberger, Kluger","year":"2019","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McInnes et al., 2018] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.","title":"Umap: Uniform manifold approximation and projection for dimension reduction","authors":"Leland Mcinnes, John Healy, James Melville","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[NENE, 1996] SA NENE. Columbia object image library (coil-100). Technical Report CUCS-006-96, 1996.","title":"SA NENE. Columbia object image library (coil-100)","authors":"Nene","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rosenberg and Hirschberg, 2007] Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In EMNLP-CoNLL, pages 410-420, 2007.","title":"V-measure: A conditional entropy-based external cluster evaluation measure","authors":"Andrew Rosenberg, Julia Hirschberg","year":"2007","venue":"EMNLP-CoNLL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sainburg et al., 2020] Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for rep- resentation and semi-supervised learning. arXiv preprint arXiv:2009.12981, 2020.","title":"Parametric umap embeddings for representation and semi-supervised learning","authors":"Sainburg","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tang et al., 2016] Jian Tang, Jingzhou Liu, Ming Zhang, and Qiaozhu Mei. Visualizing large-scale and high- dimensional data. In TheWebConf, pages 287-297, 2016.","title":"Visualizing large-scale and highdimensional data","authors":"Jian Tang, Jingzhou Liu, Ming Zhang, Qiaozhu Mei","year":"2016","venue":"TheWebConf","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tasic et al., 2018] Bosiljka Tasic, Zizhen Yao, Lucas T Graybuck, Kimberly A Smith, Thuc Nghi Nguyen, Dar- ren Bertagnolli, Jeff Goldy, Emma Garren, Michael N Economo, Sarada Viswanathan, et al. Shared and distinct transcriptomic cell types across neocortical areas. Nature, 563(7729):72-78, 2018.","title":"Shared and distinct transcriptomic cell types across neocortical areas","authors":"Tasic","year":"2018","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van Der Maaten et al., 2009] Laurens Van Der Maaten, Eric Postma, Jaap Van den Herik, et al. Dimensionality re- duction: a comparative. J Mach Learn Res, 10(66-71):13, 2009.","title":"Dimensionality reduction: a comparative","authors":"Van Der Maaten","year":"2009","venue":"J Mach Learn Res","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Van Der Maaten, 2014] Laurens Van Der Maaten. Ac- celerating t-sne using tree-based algorithms. JMLR, 15(1):3221-3245, 2014.","title":"Accelerating t-sne using tree-based algorithms","authors":"Laurens Van Der Maaten, Van Der Maaten","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021] Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how di- mension reduction tools work: an empirical approach to deciphering t-sne, umap, trimap, and pacmap for data vi- sualization. The Journal of Machine Learning Research, 22(1):9129-9201, 2021.","title":"Understanding how dimension reduction tools work: an empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization","authors":"Wang","year":"2021","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}