{"type":"ScholarlyArticle","title":"CAM-Based Methods Can See through Walls (Extended Abstract)","authors":[{"name":"Magamed Taimeskhanov","slug":"magamed-taimeskhanov","given":"Magamed","family":"Taimeskhanov","affiliation":"Julius-Maximilians Universität Würzburg , CAIDAS , Würzburg , Germany"},{"name":"Ronan Sicre","slug":"ronan-sicre","given":"Ronan","family":"Sicre","affiliation":"Centrale Méditerranée , Aix-Marseille Univ. , CNRS , LIS , Marseille , France"},{"name":"Damien Garreau","slug":"damien-garreau","given":"Damien","family":"Garreau","affiliation":"Julius-Maximilians Universität Würzburg , CAIDAS , Würzburg , Germany"}],"abstract":"CAM-based methods are widely-used post-hoc interpretability methods that produce a saliency map to explain the decision of an image classification model. The saliency map highlights the important areas of the image relevant to the prediction. In this paper, we show that most of these methods can incorrectly attribute an important score to parts of the image that the model cannot see. We show that this phenomenon occurs both theoretically and experimentally. On the theory side, we analyze the behavior of GradCAM on a simple masked CNN model at initialization. Experimentally, we train a VGG-like model constrained to not use the lower part of the image and  nevertheless observe positive scores in the unseen part of the image. This behavior is evaluated quantitatively on two new datasets. We believe that this is problematic, potentially leading to mis-interpretation of the model's behavior.","doi":"10.24963/ijcai.2025/1220","url":"https://ijcai.org/proceedings/2025/1220","pdf":"https://ijcai.org/proceedings/2025/1220.pdf","markdown":"https://ijcai.org/proceedings/2025/1220.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/1220","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":1220,"pages":{"first":10937,"last":10941},"keywords":["Sister Conferences Best Papers: Machine Learning","Sister Conferences Best Papers: Humans and AI","Sister Conferences Best Papers: Computer Vision","Sister Conferences Best Papers: AI Ethics, Trust, Fairness"],"videos":[],"references":[{"position":1,"raw":"[Adebayo et al., 2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In Advances in Neural Information Processing Systems, 2018.","title":"Sanity Checks for Saliency Maps","authors":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Benítez et al., 1997] José Manuel Benítez, Juan Luis Cas- tro, and Ignacio Requena. Are artificial neural networks black boxes? IEEE Transactions on Neural Networks, 1997.","title":"José Manuel Benítez, Juan Luis Castro, and Ignacio Requena. Are artificial neural networks black boxes","authors":"Benítez","year":"1997","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Draelos and Carin, 2021] Rachel Lea Draelos and Lawrence Carin. Use HiResCAM instead of Grad- CAM for faithful explanations of convolutional neural networks. arxiv preprint 2011.08891, 2021.","title":"Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks","authors":"Rachel Lea, Draelos, Lawrence Carin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fukushima, 1980] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biolog- ical Cybernetics, 1980.","title":"Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position","authors":"Kunihiko Fukushima","year":"1980","venue":"Biological Cybernetics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Garreau and Mardaoui, 2021] Damien Garreau and Dina Mardaoui. What does LIME really see in images? In Inter- national Conference on Machine Learning. PMLR, 2021.","title":"What does LIME really see in images?","authors":"Damien Garreau, Dina Mardaoui","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghorbani et al., 2019] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of Neural Networks is Frag- ile. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.","title":"Interpretation of Neural Networks is Fragile","authors":"Ghorbani","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Heo et al., 2019] Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling Neural Network Interpretations via Adver- sarial Model Manipulation. In Advances in Neural Infor- mation Processing Systems, 2019.","title":"Fooling Neural Network Interpretations via Adversarial Model Manipulation","authors":"Heo","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kindermans et al., 2019] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (Un)reliability of saliency methods. Explainable AI: Interpreting, explaining and visualizing deep learning, 2019.","title":"The (Un)reliability of saliency methods. Explainable AI: Interpreting, explaining and visualizing deep learning","authors":"Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, T Kristof, Sven Schütt, Dumitru Dähne, Been Erhan, Kim","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 1998.","title":"Gradient-Based Learning Applied to Document Recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Linardatos et al., 2021] Pantelis Linardatos, Vasilis Pa- pastefanopoulos, and Sotiris Kotsiantis. Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 2021.","title":"Explainable AI: A Review of Machine Learning Interpretability Methods","authors":"Pantelis Linardatos, Vasilis Papastefanopoulos, Sotiris Kotsiantis","year":"2021","venue":"Entropy","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lipton, 2018] Zachary C. Lipton. The Mythos of Model In- terpretability: In Machine Learning, the Concept of Inter- pretability is Both Important and Slippery. 2018.","title":"The Mythos of Model Interpretability: In Machine Learning, the Concept of Interpretability is Both Important and Slippery","authors":"Zachary C Lipton","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \"Why Should I Trust You?\": Ex- plaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data mining, 2016.","title":"Why Should I Trust You?\": Explaining the Predictions of Any Classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Selvaraju et al., 2017] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.","title":"Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization","authors":"R Ramprasaath, Michael Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR, 2015.","title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simonyan et al., 2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv preprint arXiv:1312.6034, 2013.","title":"Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Taimeskhanov et al., 2024] Magamed Taimeskhanov, Ro- nan Sicre, and Damien Garreau. CAM-Based Meth- ods Can See Through Walls. In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases. Springer, 2024.","title":"Magamed Taimeskhanov, Ronan Sicre, and Damien Garreau. CAM-Based Methods Can See Through Walls","authors":"Taimeskhanov","year":"2024","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yun et al., 2019] S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choe. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In IEEE/CVF International Conference on Computer Vision, 2019.","title":"CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features","authors":"S Yun, D Han, S Chun, S Oh, Y Yoo, J Choe","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang et al., 2021] Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang. A Survey on Neural Network Interpretabil- ity. IEEE Transactions on Emerging Topics in Computa- tional Intelligence, 2021.","title":"A Survey on Neural Network Interpretability","authors":"Yu Zhang, Peter Tiňo, Aleš Leonardis, Ke Tang","year":"2021","venue":"IEEE Transactions on Emerging Topics in Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2023] Hanwei Zhang, Felipe Torres, Ronan Sicre, Yannis Avrithis, and Stephane Ayache. Opti- CAM: Optimizing saliency maps for interpretability. arXiv preprint arXiv:2301.07002, 2023.","title":"Opti-CAM: Optimizing saliency maps for interpretability","authors":"Hanwei Zhang, Felipe Torres, Ronan Sicre, Yannis Avrithis, Stephane Ayache","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning Deep Features for Discriminative Localization. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"Learning Deep Features for Discriminative Localization","authors":"Zhou","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}