{"type":"ScholarlyArticle","title":"Interpretable Network Visualizations: A Human-in-the-Loop Approach for Post-hoc Explainability of CNN-based Image Classification","authors":[{"name":"Matteo Bianchi","slug":"matteo-bianchi","given":"Matteo","family":"Bianchi","affiliation":"Politecnico di Milano , DEIB , I-20133 Milano , Italy"},{"name":"Antonio De Santis","slug":"antonio-de-santis","given":"Antonio","family":"De Santis","affiliation":"Politecnico di Milano , DEIB , I-20133 Milano , Italy"},{"name":"Andrea Tocchetti","slug":"andrea-tocchetti","given":"Andrea","family":"Tocchetti","affiliation":"Politecnico di Milano , DEIB , I-20133 Milano , Italy"},{"name":"Marco Brambilla","slug":"marco-brambilla","given":"Marco","family":"Brambilla","affiliation":"Politecnico di Milano , DEIB , I-20133 Milano , Italy"}],"abstract":"Transparency and explainability in image classification are essential for establishing trust in machine learning models and detecting biases and errors. State-of-the-art explainability methods generate saliency maps to show where a specific class is identified, without providing a detailed explanation of the model's decision process. Striving to address such a need, we introduce a post-hoc method that explains the entire feature extraction process of a Convolutional Neural Network. These explanations include a layer-wise representation of the features the model extracts from the input. Such features are represented as saliency maps generated by clustering and merging similar feature maps, to which we associate a weight derived by generalizing Grad-CAM for the proposed methodology. To further enhance these explanations, we include a set of textual labels collected through a gamified crowdsourcing activity and processed using NLP techniques and Sentence-BERT. Finally, we show an approach to generate global explanations by aggregating labels across multiple images.","doi":"10.24963/ijcai.2024/411","url":"https://ijcai.org/proceedings/2024/411","pdf":"https://ijcai.org/proceedings/2024/0411.pdf","markdown":"https://ijcai.org/proceedings/2024/411.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/411","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":411,"pages":{"first":3715,"last":3723},"keywords":["Machine Learning: ML: Explainable/Interpretable machine learning","Computer Vision: CV: Interpretability and transparency","Humans and AI: HAI: Human computation and crowdsourcing"],"videos":[],"references":[{"position":1,"raw":"[Aechtner et al., 2022] Jonathan Aechtner, Lena Cabrera, Dennis Katwal, Pierre Onghena, Diego Penroz Valenzuela, and Anna Wilbik. Comparing user perception of explana- tions developed with xai methods. In 2022 IEEE Interna- tional Conference on Fuzzy Systems (FUZZ-IEEE), pages 1-7, 2022.","title":"Comparing user perception of explanations developed with xai methods","authors":"Jonathan Aechtner, Lena Cabrera, Dennis Katwal, Pierre Onghena, Diego Penroz Valenzuela, Anna Wilbik","year":"2022","venue":"IEEE International Conference on Fuzzy Systems (FUZZ-IEEE","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arrieta et al., 2019] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil- López, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai, 2019.","title":"Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai","authors":"Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, Raja Chatila, Francisco Herrera","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balayn et al., 2021] Agathe Balayn, Panagiotis Soilis, Christoph Lofi, Jie Yang, and Alessandro Bozzon. What do you mean? interpreting image classification with crowdsourced concept extraction and analysis. In Pro- ceedings of the Web Conference 2021, WWW '21, page 1937-1948, New York, NY, USA, 2021. Association for Computing Machinery.","title":"What do you mean? interpreting image classification with crowdsourced concept extraction and analysis","authors":"Agathe Balayn, Panagiotis Soilis, Christoph Lofi, Jie Yang, Alessandro Bozzon","year":"2021","venue":"Proceedings of the Web Conference 2021","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brooke, 1995] John Brooke. Sus: A quick and dirty usabil- ity scale. Usability Eval. Ind., 189, 11 1995.","title":"Sus: A quick and dirty usability scale","authors":"John Brooke","year":"1995","venue":"Usability Eval. Ind","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brooke, 2013] John Brooke. Sus: a retrospective. Journal of Usability Studies, 8:29-40, 01 2013.","title":"John Brooke. Sus: a retrospective","authors":"Brooke","year":"2013","venue":"Journal of Usability Studies","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Erion et al., 2019] Gabriel G. Erion, Joseph D. Janizek, Pascal Sturmfels, Scott M. Lundberg, and Su-In Lee. Learning explainable models using attribution priors. CoRR, abs/1906.10670, 2019.","title":"Learning explainable models using attribution priors","authors":"G Gabriel, Joseph D Erion, Pascal Janizek, Scott M Sturmfels, Su-In Lundberg, Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Estivill-Castro et al., 2020] Vladimir Estivill-Castro, Eu- gene Gilmore, and René Hexel. Human-in-the-loop con- struction of decision tree classifiers with parallel coordi- nates. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 3852-3859, 2020.","title":"Human-in-the-loop construction of decision tree classifiers with parallel coordinates","authors":"Vladimir Estivill-Castro, Eugene Gilmore, René Hexel","year":"2020","venue":"IEEE International Conference on Systems, Man, and Cybernetics (SMC)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guidotti et al., 2018] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Dino Pedreschi, and Fosca Giannotti. A survey of methods for explaining black box models, 2018.","title":"A survey of methods for explaining black box models","authors":"Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Dino Pedreschi, Fosca Giannotti","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hart and Staveland, 1988] Sandra G. Hart and Lowell E. Staveland. Development of nasa-tlx (task load index): Re- sults of empirical and theoretical research. In Peter A. Hancock and Najmedin Meshkati, editors, Human Mental Workload, volume 52 of Advances in Psychology, pages 139-183. North-Holland, 1988.","title":"Development of nasa-tlx (task load index): Results of empirical and theoretical research","authors":"Sandra G Hart, Lowell E Staveland","year":"1988","venue":"Human Mental Workload","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2017] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). 2017.","title":"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors","authors":"Kim","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Neural Information Pro- cessing Systems, 25, 01 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton","year":"2012","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lipton, 2016] Zachary Chase Lipton. The mythos of model interpretability. CoRR, abs/1606.03490, 2016.","title":"The mythos of model interpretability","authors":"Zachary Chase, Lipton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2021] Xiaotian Lu, Arseny Tolmachev, Tatsuya Yamamoto, Koh Takeuchi, Seiji Okajima, Tomoyoshi Takebayashi, Koji Maruhashi, and Hisashi Kashima. Crowdsourcing evaluation of saliency-based XAI meth- ods. CoRR, abs/2107.00456, 2021.","title":"Crowdsourcing evaluation of saliency-based XAI methods","authors":"Xiaotian Lu, Arseny Tolmachev, Tatsuya Yamamoto, Koh Takeuchi, Seiji Okajima, Tomoyoshi Takebayashi, Koji Maruhashi, Hisashi Kashima","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lundberg and Lee, 2017] Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions, 2017.","title":"A unified approach to interpreting model predictions","authors":"Lee ; Lundberg, Scott Lundberg, Su-In Lee","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Magister et al., 2021] Lucie Charlotte Magister, Dmitry Kazhdan, Vikash Singh, and Pietro Liò. Gcexplainer: Human-in-the-loop concept-based explanations for graph neural networks. CoRR, abs/2107.11889, 2021.","title":"Gcexplainer: Human-in-the-loop concept-based explanations for graph neural networks","authors":"Lucie Charlotte Magister, Dmitry Kazhdan, Vikash Singh, Pietro Liò","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maćkiewicz and Ratajczak, 1993] Andrzej Maćkiewicz and Waldemar Ratajczak. Principal components analysis (pca). Computers & Geosciences, 19(3):303-342, 1993.","title":"Principal components analysis (pca)","authors":"Andrzej Maćkiewicz, Waldemar Ratajczak","year":"1993","venue":"Computers & Geosciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mishra and Rzeszotarski, 2021] Swati Mishra and Jef- frey M. Rzeszotarski. Crowdsourcing and evaluating concept-driven explanations of machine learning models. Proc. ACM Hum.-Comput. Interact., 5(CSCW1), apr 2021.","title":"Crowdsourcing and evaluating concept-driven explanations of machine learning models","authors":"Swati Mishra, Jeffrey M Rzeszotarski","year":"2021","venue":"Proc. ACM Hum.-Comput. Interact","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Morschheuser et al., 2016] Benedikt Morschheuser, Juho Hamari, and Jonna Koivisto. Gamification in crowdsourc- ing: A review. 01 2016.","title":"Gamification in crowdsourcing: A review","authors":"Benedikt Morschheuser, Juho Hamari, Jonna Koivisto","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Prabaswari et al., 2019] Atyanti Dyah Prabaswari, Chan- card Basumerda, and Bagus Wahyu Utomo. The mental workload analysis of staff in study program of private ed- ucational organization. IOP Conference Series: Materials Science and Engineering, 528(1):012018, may 2019.","title":"The mental workload analysis of staff in study program of private educational organization","authors":"Prabaswari","year":"2019","venue":"IOP Conference Series: Materials Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. CoRR, abs/1908.10084, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \"why should i trust you?\": Explain- ing the predictions of any classifier, 2016.","title":"why should i trust you?\": Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Selvaraju et al., 2016] Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient- based localization. CoRR, abs/1610.02391, 2016.","title":"Grad-cam: Why did you say that? visual explanations from deep networks via gradientbased localization","authors":"R Ramprasaath, Abhishek Selvaraju, Ramakrishna Das, Michael Vedantam, Devi Cogswell, Dhruv Parikh, Batra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan et al., 2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013.","title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sundararajan et al., 2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep net- works. CoRR, abs/1703.01365, 2017.","title":"Axiomatic attribution for deep networks","authors":"Mukund Sundararajan, Ankur Taly, Qiqi Yan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Szymanski et al., 2021] Maxwell Szymanski, Martijn Mil- lecamp, and Katrien Verbert. Visual, textual or hybrid: The effect of user expertise on different explanations. In 26th International Conference on Intelligent User Inter- faces, IUI '21, page 109-119, New York, NY, USA, 2021. Association for Computing Machinery.","title":"Visual, textual or hybrid: The effect of user expertise on different explanations","authors":"Maxwell Szymanski, Martijn Millecamp, Katrien Verbert","year":"2021","venue":"26th International Conference on Intelligent User Interfaces, IUI '21","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tocchetti and Brambilla, 2022] Andrea Tocchetti and Marco Brambilla. The role of human knowledge in explainable ai. Data, 7(7), 2022.","title":"The role of human knowledge in explainable ai","authors":"Brambilla Tocchetti, Andrea Tocchetti, Marco Brambilla","year":"2022","venue":"Data","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tocchetti et al., 2022] Andrea Tocchetti, Lorenzo Corti, Marco Brambilla, and Irene Celino. Exp-crowd: A gam- ified crowdsourcing framework for explainability. Fron- tiers in Artificial Intelligence, 5, 2022.","title":"Exp-crowd: A gamified crowdsourcing framework for explainability","authors":"Andrea Tocchetti, Lorenzo Corti, Marco Brambilla, Irene Celino","year":"2022","venue":"Frontiers in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tong and Kagal, 2020] Schrasing Tong and Lalana Kagal. Investigating bias in image classification using model ex- planations, 2020.","title":"Investigating bias in image classification using model explanations","authors":"Kagal Tong, Schrasing Tong, Lalana Kagal","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[van der Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 11 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[von Eschenbach, 2021] Warren von Eschenbach. Trans- parency and the black box problem: Why we do not trust ai. Philosophy & Technology, 34, 12 2021.","title":"Warren von Eschenbach. Transparency and the black box problem: Why we do not trust ai","authors":"Von Eschenbach","year":"2021","venue":"Philosophy & Technology","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2019] Jun Wang, Changsheng Zhao, Junfu Xi- ang, and Kanji Uchino. Interactive topic model with en- hanced interpretability. In IUI Workshops, 2019.","title":"Interactive topic model with enhanced interpretability","authors":"Wang","year":"2019","venue":"IUI Workshops","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2019] Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu. Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges, pages 563-574. 09 2019.","title":"Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges","authors":"Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, Jun Zhu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zeiler and Fergus, 2013] Matthew D. Zeiler and Rob Fer- gus. Visualizing and understanding convolutional net- works. CoRR, abs/1311.2901, 2013.","title":"Visualizing and understanding convolutional networks","authors":"Matthew D Zeiler, Rob Fergus","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}