{"type":"ScholarlyArticle","title":"ViT-CX: Causal Explanation of Vision Transformers","authors":[{"name":"Weiyan Xie","slug":"weiyan-xie","given":"Weiyan","family":"Xie","affiliation":"The Hong Kong University of Science and Technology , China"},{"name":"Xiao-Hui Li","slug":"xiao-hui-li","given":"Xiao-Hui","family":"Li","affiliation":"Huawei Technologies Co. , Ltd , China"},{"name":"Caleb Chen Cao","slug":"caleb-chen-cao","given":"Caleb Chen","family":"Cao","affiliation":"The Hong Kong University of Science and Technology , China"},{"name":"Nevin L. Zhang","slug":"nevin-l-zhang","given":"Nevin L.","family":"Zhang","affiliation":"The Hong Kong University of Science and Technology , China"}],"abstract":"Despite the popularity of Vision Transformers (ViTs) and eXplainable AI (XAI), only a few explanation methods have been designed specially for ViTs thus far. They mostly use attention weights of the [CLS] token on patch embeddings and often produce unsatisfactory saliency maps. This paper proposes a novel method for explaining ViTs called ViT-CX. It is based on patch embeddings, rather than attentions paid to them, and their causal impacts on the model output. Other characteristics of ViTs such as causal overdetermination are considered in the design of ViT-CX. The empirical results show that ViT-CX produces more meaningful saliency maps and does a better job revealing all important evidence for the predictions than previous methods. The explanation generated by ViT-CX also shows significantly better faithfulness to the model. The codes and appendix are available at https://github.com/vaynexie/CausalX-ViT.","doi":"10.24963/ijcai.2023/174","url":"https://ijcai.org/proceedings/2023/174","pdf":"https://ijcai.org/proceedings/2023/0174.pdf","markdown":"https://ijcai.org/proceedings/2023/174.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/174","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":174,"pages":{"first":1569,"last":1577},"keywords":["Computer Vision: CV: Interpretability and transparency","Machine Learning: ML: Explainable/Interpretable machine learning","AI Ethics, Trust, Fairness: ETF: Explainability and interpretability"],"videos":[],"references":[{"position":1,"raw":"[Abnar and Zuidema, 2020] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, 2020.","title":"Quantifying attention flow in transformers","authors":"Zuidema Abnar, Samira Abnar, Willem Zuidema","year":"2020","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Adebayo et al., 2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Ad- vances in Neural Information Processing Systems, pages 9525-9536, 2018.","title":"Sanity checks for saliency maps","authors":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bach et al., 2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.","title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","authors":"Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, Wojciech Samek","year":"2015","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bastings and Filippova, 2020] Jasmijn Bastings and Katja Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency meth- ods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149-155, 2020.","title":"The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?","authors":"Filippova Bastings, Jasmijn Bastings, Katja Filippova","year":"2020","venue":"Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chefer et al., 2021a] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi- modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397-406, 2021.","title":"Generic attention-model explainability for interpreting bimodal and encoder-decoder transformers","authors":"Chefer","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chefer et al., 2021b] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualiza- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782-791, 2021.","title":"Transformer interpretability beyond attention visualization","authors":"Chefer","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chu et al., 2021] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural In- formation Processing Systems, 34:9355-9366, 2021.","title":"Twins: Revisiting the design of spatial attention in vision transformers","authors":"Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, Chunhua Shen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Covert et al., 2022] Ian Covert, Chanwoo Kim, and Su-In Lee. Learning to estimate shapley values with vision trans- formers. arXiv preprint arXiv:2206.05282, 2022.","title":"and Su-In Lee. Learning to estimate shapley values with vision transformers","authors":"Covert","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.","title":"Imagenet: A largescale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Doshi-Velez and Kim, 2017] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.","title":"Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning","authors":"Kim Doshi-Velez","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International conference on learning represen- tations, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"International conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jain and Wallace, 2019] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of NA Annual Meeting of the Association for Computational Linguistics- HLT, pages 3543-3556, 2019.","title":"Attention is not explanation","authors":"Sarthak Jain, Byron C Wallace","year":"2019","venue":"Proceedings of NA Annual Meeting of the Association for Computational Linguistics-HLT","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Müllner, 2013] Daniel Müllner. fastcluster: Fast hierarchi- cal, agglomerative clustering routines for r and python. Journal of Statistical Software, 53:1-18, 2013.","title":"fastcluster: Fast hierarchical, agglomerative clustering routines for r and python","authors":"Daniel Müllner","year":"2013","venue":"Journal of Statistical Software","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Murtagh and Legendre, 2014] Fionn Murtagh and Pierre Legendre. Ward's hierarchical agglomerative clustering method: which algorithms implement ward's criterion? Journal of classification, 31(3):274-295, 2014.","title":"Fionn Murtagh and Pierre Legendre. Ward's hierarchical agglomerative clustering method: which algorithms implement ward's criterion?","authors":"Legendre Murtagh","year":"2014","venue":"Journal of classification","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Naseer et al., 2021] Muzammal Naseer, Kanchana Ranas- inghe, Salman Khan, Munawar Hayat, Fahad Khan, and Ming-Hsuan Yang. Intriguing properties of vision trans- formers. In Advances in Neural Information Processing Systems, 2021.","title":"Intriguing properties of vision transformers","authors":"Naseer","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Petsiuk et al., 2018] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explana- tion of black-box models. In Proceedings of the British Machine Vision Conference, 2018.","title":"Rise: Randomized input sampling for explanation of black-box models","authors":"Abir Vitali Petsiuk, Kate Das, Saenko","year":"2018","venue":"Proceedings of the British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pruthi et al., 2020] Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. Learn- ing to deceive with attention-based explanations. In An- nual Meeting of the Association for Computational Lin- guistics, pages 4782-4793, 2020.","title":"Learning to deceive with attention-based explanations","authors":"Pruthi","year":"2020","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \" why should i trust you?\" explain- ing the predictions of any classifier. In Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.","title":"why should i trust you?\" explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sattarzadeh et al., 2021] Sam Sattarzadeh, Mahesh Sud- hakar, Anthony Lem, Shervin Mehryar, Konstantinos N Plataniotis, Jongseong Jang, Hyunwoo Kim, Yeonjeong Jeong, Sangmin Lee, and Kyunghoon Bae. Explaining convolutional neural networks through attribution-based input sampling and block-wise feature aggregation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 11639-11647, 2021.","title":"Explaining convolutional neural networks through attribution-based input sampling and block-wise feature aggregation","authors":"Sam Sattarzadeh, Mahesh Sudhakar, Anthony Lem, Shervin Mehryar, N Konstantinos, Jongseong Plataniotis, Hyunwoo Jang, Yeonjeong Kim, Sangmin Jeong, Kyunghoon Lee, Bae","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Serrano and Smith, 2019] Sofia Serrano and Noah A Smith. Is attention interpretable? In Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2931-2951, 2019.","title":"Sofia Serrano and Noah A Smith. Is attention interpretable?","authors":"Smith Serrano","year":"2019","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Smilkov et al., 2017] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.","title":"Smoothgrad: removing noise by adding noise","authors":"Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, Martin Wattenberg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sundararajan et al., 2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep net- works. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.","title":"Axiomatic attribution for deep networks","authors":"Mukund Sundararajan, Ankur Taly, Qiqi Yan","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In In- ternational Conference on Machine Learning, pages 10347-10357. PMLR, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ternational Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tuli et al., 2021] Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Tom Griffiths. Are convolutional neural networks or transformers more like human vision? In Proceedings of the Annual Meeting of the Cognitive Science Society, vol- ume 43, 2021.","title":"Are convolutional neural networks or transformers more like human vision","authors":"Shikhar Tuli, Ishita Dasgupta, Erin Grant, Tom Griffiths","year":"2021","venue":"Proceedings of the Annual Meeting of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Voita et al., 2019] Elena Voita, David Talbot, Fedor Moi- seev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Annual Meeting of the Association for Computational Linguistics, pages 5797-5808. Annual Meeting of the Association for Computational Linguistics Anthology, 2019.","title":"Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned","authors":"Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov","year":"2019","venue":"Annual Meeting of the Association for Computational Linguistics Anthology","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2020] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explana- tions for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshop, pages 24-25, 2020.","title":"Score-cam: Score-weighted visual explanations for convolutional neural networks","authors":"Wang","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshop","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[White et al., 2021] Adam White, Kwun Ho Ngan, James Phelan, Saman Sadeghi Afgeh, Kevin Ryan, Con- stantino Carlos Reyes-Aldasoro, and Artur d'Avila Garcez. Contrastive counterfactual visual explanations with overdetermination. arXiv preprint arXiv:2106.14556, 2021.","title":"Contrastive counterfactual visual explanations with overdetermination","authors":"Adam White, Kwun Ho Ngan, James Phelan, Saman Sadeghi Afgeh, Kevin Ryan, Constantino Carlos Reyes-Aldasoro, Artur D'avila Garcez","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yuan et al., 2021] Tingyi Yuan, Xuhong Li, Haoyi Xiong, Hui Cao, and Dejing Dou. Explaining information flow inside vision transformers using markov chain. In eXplain- able AI approaches for debugging and diagnosis., 2021.","title":"Explaining information flow inside vision transformers using markov chain","authors":"Tingyi Yuan, Xuhong Li, Haoyi Xiong, Hui Cao, Dejing Dou","year":"2021","venue":"eXplainable AI approaches for debugging and diagnosis","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2018] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation back- prop. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, 126(10):1084-1102, 2018.","title":"Top-down neural attention by excitation backprop","authors":"Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, Stan Sclaroff","year":"2018","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2022] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Sercan Ö Arik, and Tomas Pfister. Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3417-3425, 2022.","title":"Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding","authors":"Zhang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}