{"type":"ScholarlyArticle","title":"Temporality Spatialization: A Scalable and Faithful Time-Travelling Visualization for Deep Classifier Training","authors":[{"name":"Xianglin Yang","slug":"xianglin-yang","given":"Xianglin","family":"Yang","affiliation":"Department of Computer Science , Shanghai Jiao Tong University , China; School of Computing , National University of Singapore , Singapore"},{"name":"Yun Lin","slug":"yun-lin","given":"Yun","family":"Lin","affiliation":"Department of Computer Science , Shanghai Jiao Tong University , China; School of Computing , National University of Singapore , Singapore"},{"name":"Ruofan Liu","slug":"ruofan-liu","given":"Ruofan","family":"Liu","affiliation":"School of Computing , National University of Singapore , Singapore"},{"name":"Jin Song Dong","slug":"jin-song-dong","given":"Jin Song","family":"Dong","affiliation":"School of Computing , National University of Singapore , Singapore"}],"abstract":"Time-travelling visualization answers how the predictions of a deep classifier are formed during the training. It visualizes in two or three dimensional space how the classification boundaries and sample embeddings are evolved during training. \r\n\r\nIn this work, we propose TimeVis, a novel time-travelling visualization solution for deep classifiers. Comparing to the state-of-the-art solution DeepVisualInsight (DVI), TimeVis can significantly (1) reduce visualization errors for rendering samples’ travel across different training epochs, and (2) improve the visualization efficiency. To this end, we design a technique called temporality spatialization, which unifies the spatial relation (e.g., neighbouring samples in single epoch) and temporal relation (e.g., one identical sample in neighbouring training epochs) into one high-dimensional topological complex. Such spatio-temporal complex can be used to efficiently train one visualization model to accurately project and inverse-project any high and low dimensional data across epochs. Our extensive experiment shows that, in comparison to DVI, TimeVis not only is more accurate to preserve the visualized time-travelling semantics, but 15X faster in visualization efficiency, achieving a new state-of-the-art in time-travelling visualization.","doi":"10.24963/ijcai.2022/558","url":"https://ijcai.org/proceedings/2022/558","pdf":"https://ijcai.org/proceedings/2022/0558.pdf","markdown":"https://ijcai.org/proceedings/2022/558.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/558","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":558,"pages":{"first":4022,"last":4028},"keywords":["AI Ethics, Trust, Fairness: Explainability and Interpretability","Multidisciplinary Topics and Applications: Software Engineering"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986894","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985037","length_sec":343,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Adebayo et al., 2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03292, 2018.","title":"Sanity checks for saliency maps","authors":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alsallakh et al., 2021] Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Shubham Muttepawar, Edward Wang, Sara Zhang, David Adkins, and Orion Reblitz-Richardson. De- bugging the internals of convolutional networks. In eX- plainable AI approaches for debugging and diagnosis., 2021.","title":"Debugging the internals of convolutional networks","authors":"Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Shubham Muttepawar, Edward Wang, Sara Zhang, David Adkins, Orion Reblitz-Richardson","year":"2021","venue":"eXplainable AI approaches for debugging and diagnosis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chattopadhyay et al., 2019] Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vineeth N Balasubrama- nian. Neural network attributions: A causal perspective. In International Conference on Machine Learning, pages 981-990. PMLR, 2019.","title":"Neural network attributions: A causal perspective","authors":"Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, Vineeth N Balasubramanian","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chazal et al., 2014] Frédéric Chazal, Vin De Silva, and Steve Oudot. Persistence stability for geometric com- plexes. Geometriae Dedicata, 173(1):193-214, 2014.","title":"Persistence stability for geometric complexes","authors":"Frédéric Chazal, Vin De Silva, Steve Oudot","year":"2014","venue":"Geometriae Dedicata","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chazal et al., 2015] Frederic Chazal, Brittany Fasy, Fab- rizio Lecci, Bertrand Michel, Alessandro Rinaldo, and Larry Wasserman. Subsampling methods for persistent ho- mology. In Francis Bach and David Blei, editors, ICML, volume 37 of Proceedings of Machine Learning Research, pages 2143-2151, Lille, France, 07-09 Jul 2015. PMLR.","title":"Subsampling methods for persistent homology","authors":"Frederic Chazal, Brittany Fasy, Fabrizio Lecci, Bertrand Michel, Alessandro Rinaldo, Larry Wasserman","year":"2015","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chazal et al., 2016] Frédéric Chazal, Vin De Silva, Marc Glisse, and Steve Oudot. The structure and stability of persistence modules. Springer, 2016.","title":"The structure and stability of persistence modules","authors":"Frédéric Chazal, Vin De Silva, Marc Glisse, Steve Oudot","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Facco et al., 2017] Elena Facco, Maria d'Errico, Alex Ro- driguez, and Alessandro Laio. Estimating the intrinsic di- mension of datasets by a minimal neighborhood informa- tion. Scientific reports, 7(1):1-8, 2017.","title":"Estimating the intrinsic dimension of datasets by a minimal neighborhood information","authors":"Elena Facco, Maria Errico, Alex Rodriguez, Alessandro Laio","year":"2017","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farahani and Hekmatfar, 2009] Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.","title":"Facility location: concepts, models, algorithms and case studies","authors":"Reza Zanjirani, Farahani, Masoud Hekmatfar","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gonzalez, 1985] Teofilo F Gonzalez. Clustering to mini- mize the maximum intercluster distance. Theoretical com- puter science, 38:293-306, 1985.","title":"Clustering to minimize the maximum intercluster distance","authors":"F Teofilo, Gonzalez","year":"1985","venue":"Theoretical computer science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Graving and Couzin, 2020] Jacob M Graving and Iain D Couzin. Vae-sne: a deep generative model for simulta- neous dimensionality reduction and clustering. BioRxiv, 2020.","title":"Vae-sne: a deep generative model for simultaneous dimensionality reduction and clustering","authors":"Couzin ; Graving, M Jacob, Iain D Graving, Couzin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2017] Yun Lin, Jun Sun, Yinxing Xue, Yang Liu, and Jinsong Dong. Feedback-based debugging. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pages 393-403. IEEE, 2017.","title":"Feedback-based debugging","authors":"Yun Lin, Jun Sun, Yinxing Xue, Yang Liu, Jinsong Dong","year":"2017","venue":"IEEE/ACM 39th International Conference on Software Engineering (ICSE)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mallick et al., 2019] Koushik Mallick, Sanghamitra Bandy- opadhyay, Subhasis Chakraborty, Rounaq Choudhuri, and Sayan Bose. Topo2vec: A novel node embedding gen- eration based on network topology for link prediction. IEEE Transactions on Computational Social Systems, 6(6):1306-1317, 2019.","title":"Topo2vec: A novel node embedding generation based on network topology for link prediction","authors":"Koushik Mallick, Sanghamitra Bandyopadhyay, Subhasis Chakraborty, Rounaq Choudhuri, Sayan Bose","year":"2019","venue":"IEEE Transactions on Computational Social Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McInnes et al., 2018] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.","title":"Umap: Uniform manifold approximation and projection for dimension reduction","authors":"Leland Mcinnes, John Healy, James Melville","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Moor et al., 2020] Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pages 7045-7054. PMLR, 2020.","title":"Topological autoencoders","authors":"Michael Moor, Max Horn, Bastian Rieck, Karsten Borgwardt","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sainburg et al., 2021] Tim Sainburg, Leland McInnes, and Timothy Q. Gentner. Parametric UMAP Embeddings for Representation and Semisupervised Learning. Neural Computation, 33(11):2881-2907, 10 2021.","title":"Parametric UMAP Embeddings for Representation and Semisupervised Learning","authors":"Tim Sainburg, Leland Mcinnes, Timothy Q Gentner","year":"2021","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sener and Savarese, 2017] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core- set approach. arXiv preprint arXiv:1708.00489, 2017.","title":"Active learning for convolutional neural networks: A coreset approach","authors":"Ozan Sener, Silvio Savarese","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sundararajan et al., 2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep net- works. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.","title":"Axiomatic attribution for deep networks","authors":"Mukund Sundararajan, Ankur Taly, Qiqi Yan","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2019] Haijun Wang, Yun Lin, Zijiang Yang, Jun Sun, Yang Liu, Jin Song Dong, Qinghua Zheng, and Ting Liu. Explaining regressions via alignment slicing and mending. IEEE Transactions on Software Engineering, 2019.","title":"Explaining regressions via alignment slicing and mending","authors":"Haijun Wang, Yun Lin, Zijiang Yang, Jun Sun, Yang Liu, Jin Song Dong, Qinghua Zheng, Ting Liu","year":"2019","venue":"IEEE Transactions on Software Engineering","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiao et al., 2021] Yan Xiao, Ivan Beschastnikh, David S Rosenblum, Changsheng Sun, Sebastian Elbaum, Yun Lin, and Jin Song Dong. Self-checking deep neural networks in deployment. In 2021 IEEE/ACM 43rd International Con- ference on Software Engineering (ICSE), pages 372-384. IEEE, 2021.","title":"Self-checking deep neural networks in deployment","authors":"Yan Xiao, Ivan Beschastnikh, David S Rosenblum, Changsheng Sun, Sebastian Elbaum, Yun Lin, Jin Song, Dong","year":"2021","venue":"IEEE/ACM 43rd International Conference on Software Engineering (ICSE)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2022a] Xianglin Yang, Yun Lin, Ruofan Liu, and Jin Song Dong. Appendix. https://sites.google.com/ view/timevis/home, 2022. Accessed: 2022-01-15.","title":"Appendix","authors":"Yang","year":"2022","venue":null,"doi":null,"url":"https://sites.google.com/view/timevis/home","internal_url":null},{"position":21,"raw":"[Yang et al., 2022b] Xianglin Yang, Yun Lin, Ruofan Liu, and Jin Song Dong. Website of timevis. https://sites. google.com/view/timevis/home, 2022. Accessed: 2022- 01-15.","title":"Website of timevis","authors":"Yang","year":"2022","venue":null,"doi":null,"url":"https://sites.google.com/view/timevis/home,2022","internal_url":null},{"position":22,"raw":"[Yang et al., 2022c] Xianglin Yang, Yun Lin, Ruofan Liu, Zhenfeng He, Chao Wang, Jin Song Dong, and Hong Mei. Deepvisualinsight: Time-travelling visualization for spatio-temporal causality of deep classification training. In The Thirty-Sixth AAAI Conference on Artificial Intel- ligence (AAAI), 2022.","title":"Deepvisualinsight: Time-travelling visualization for spatio-temporal causality of deep classification training","authors":"Yang","year":"2022","venue":"The Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}