{"type":"ScholarlyArticle","title":"Libra-CAM: An Activation-Based Attribution Based on the Linear Approximation of Deep Neural Nets and Threshold Calibration","authors":[{"name":"Sangkyun Lee","slug":"sangkyun-lee","given":"Sangkyun","family":"Lee","affiliation":"School of Cybersecurity , Korea University"},{"name":"Sungmin Han","slug":"sungmin-han","given":"Sungmin","family":"Han","affiliation":"School of Cybersecurity , Korea University"}],"abstract":"Universal application of AI has increased the need to explain why an AI model makes a specific decision in a human-understandable form. Among many related works, the class activation map (CAM)-based methods have been successful recently, creating input attribution based on the weighted sum of activation maps in convolutional neural networks. However, existing methods use channel-wise importance weights with specific architectural assumptions, relying on arbitrarily chosen attribution threshold values in their quality assessment: we think these can degrade the quality of attribution. In this paper, we propose Libra-CAM, a new CAM-style attribution method based on the best linear approximation of the layer (as a function) between the penultimate activation and the target-class score output. From the approximation, we derive the base formula of Libra-CAM, which is applied with multiple reference activations from a pre-built library. We construct Libra-CAM by averaging these base attribution maps, taking a threshold calibration procedure to optimize its attribution quality. Our experiments show that Libra-CAM can be computed in a reasonable time and is superior to the existing attribution methods in quantitative and qualitative attribution quality evaluations.","doi":"10.24963/ijcai.2022/442","url":"https://ijcai.org/proceedings/2022/442","pdf":"https://ijcai.org/proceedings/2022/0442.pdf","markdown":"https://ijcai.org/proceedings/2022/442.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/442","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":442,"pages":{"first":3185,"last":3191},"keywords":["AI Ethics, Trust, Fairness: Trustworthy AI","Machine Learning: Explainable/Interpretable Machine Learning","AI Ethics, Trust, Fairness: Explainability and Interpretability","Computer Vision: Interpretability and Transparency"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986812","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984955","length_sec":331,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Adebayo et al., 2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In NIPS, volume 31, 2018.","title":"Sanity checks for saliency maps","authors":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bach et al., 2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wo- jciech Samek. On pixel-wise explanations for non-linear classi- fier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-46, 2015.","title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","authors":"Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, Wojciech Samek","year":"2015","venue":"PLOS ONE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balduzzi et al., 2017] David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In ICML, pages 342-350, 2017.","title":"The shattered gradients problem: If resnets are the answer, then what is the question?","authors":"David Balduzzi, Marcus Frean, Lennox Leary, J P Lewis, Kurt, Wan-Duo Ma, Brian Mcwilliams","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chattopadhay et al., 2018] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In WACV, pages 839-847, 2018.","title":"Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks","authors":"Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, Vineeth N Balasubramanian","year":"2018","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chouldechova and Roth, 2020] Alexandra Chouldechova and Aaron Roth. A snapshot of the frontiers of fairness in machine learning. Commun. ACM, 63(5):82-89, 2020.","title":"Alexandra Chouldechova and Aaron Roth. A snapshot of the frontiers of fairness in machine learning","authors":"Roth Chouldechova","year":"2020","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Courant and John, 1965] Richard Courant and Fritz John. Intro- duction to Calculus and Analysis, volume 1. Interscience Pub- lishers, 1965.","title":"Introduction to Calculus and Analysis","authors":"Richard Courant, Fritz John","year":"1965","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Erhan et al., 2009] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report 1341, University of Montreal, 2009.","title":"Visualizing higher-layer features of a deep network","authors":"Yoshua Dumitru Erhan, Aaron Bengio, Pascal Courville, Vincent","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fong and Vedaldi, 2017] Ruth C. Fong and Andrea Vedaldi. Inter- pretable explanations of black boxes by meaningful perturbation. In ICCV, 2017.","title":"Interpretable explanations of black boxes by meaningful perturbation","authors":"Ruth C Fong, Andrea Vedaldi","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[GDPR, 2016] GDPR. EU General Data Protection Regulation OJ 2016 L 119/1. European Commission, 2016.","title":"GDPR. EU General Data Protection Regulation OJ","authors":"Gdpr","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2018] Jindong Gu, Yinchong Yang, and Volker Tresp. Understanding individual decisions of cnns via contrastive back- propagation. In ACCV, pages 119-134, 2018.","title":"Understanding individual decisions of cnns via contrastive backpropagation","authors":"Jindong Gu, Yinchong Yang, Volker Tresp","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gunning, 2016] David Gunning. Explainable artificial intelligence (XAI), DARPA-BAA-16-53. DARPA, 2016.","title":"Explainable artificial intelligence (XAI)","authors":"David Gunning","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gur et al., 2021] Shir Gur, Ameen Ali, and Lior Wolf. Visualiza- tion of supervised and self-supervised neural networks via attri- bution guided factorization. In AAAI, pages 11545-11554, 2021.","title":"Visualization of supervised and self-supervised neural networks via attribution guided factorization","authors":"Shir Gur, Ameen Ali, Lior Wolf","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiménez-Luna et al., 2020] José Jiménez-Luna, Francesca Grisoni, and Gisbert Schneider. Drug discovery with ex- plainable artificial intelligence. Nature Machine Intelligence, 2(10):573-584, 2020.","title":"Drug discovery with explainable artificial intelligence","authors":"Jiménez-Luna","year":"2020","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kapishnikov et al., 2019] Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viegas, and Michael Terry. XRAI: Better attributions through regions. In ICCV, pages 4947-4956, 2019.","title":"XRAI: Better attributions through regions","authors":"Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viegas, Michael Terry","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kapishnikov et al., 2021] Andrei Kapishnikov, Subhashini Venu- gopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. In CVPR, pages 5050-5058, 2021.","title":"Guided integrated gradients: An adaptive path method for removing noise","authors":"Kapishnikov","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2021] Jeong Ryong Lee, Sewon Kim, Inyong Park, Tae- joon Eo, and Dosik Hwang. Relevance-cam: Your model already knows where to look. In CVPR, pages 14944-14953, 2021.","title":"Relevance-cam: Your model already knows where to look","authors":"Jeong Ryong Lee, Sewon Kim, Inyong Park, Taejoon Eo, Dosik Hwang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lundberg and Lee, 2017] Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NIPS, volume 30, 2017.","title":"A unified approach to interpreting model predictions","authors":"Lee ; Lundberg, Scott Lundberg, Su-In Lee","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nam et al., 2020] Woo-Jeoung Nam, Jaesik Choi, and Seong- Whan Lee. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural net- works. In AAAI, 2020.","title":"Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks","authors":"Nam","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nocedal and Wright, 2006] Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer, second edition, 2006.","title":"Numerical Optimization","authors":"Jorge Nocedal, Stephen Wright","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oquab et al., 2014] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image repre- sentations using convolutional neural networks. In CVPR, 2014.","title":"Learning and transferring mid-level image representations using convolutional neural networks","authors":"Maxime Oquab, Leon Bottou, Ivan Laptev, Josef Sivic","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Petsiuk et al., 2018] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In BMVC, 2018.","title":"Rise: Randomized input sampling for explanation of black-box models","authors":"Abir Vitali Petsiuk, Kate Das, Saenko","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qi et al., 2020] Zhongang Qi, Saeed Khorram, and Fuxin Li. Vi- sualizing deep networks by optimizing with integrated gradients. In AAAI, pages 11890-11898, 2020.","title":"Visualizing deep networks by optimizing with integrated gradients","authors":"Qi","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Car- los Guestrin. \"Why should i trust you?\": Explaining the predic- tions of any classifier. In KDD, pages 1135-1144, 2016.","title":"Why should i trust you?\": Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Selvaraju et al., 2017] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Selvaraju","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shrikumar et al., 2017] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In ICML, pages 3145-3153, 2017.","title":"Learning important features through propagating activation differences","authors":"Shrikumar","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Springenberg et al., 2015] Jost Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop), 2015.","title":"Striving for simplicity: The all convolutional net","authors":"Jost Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin Riedmiller","year":"2015","venue":"ICLR (workshop)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sundararajan et al., 2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, pages 3319-3328, 2017.","title":"Axiomatic attribution for deep networks","authors":"Mukund Sundararajan, Ankur Taly, Qiqi Yan","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2020] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolu- tional neural networks. In CVPR (workshop), pages 24-25, 2020.","title":"Score-cam: Score-weighted visual explanations for convolutional neural networks","authors":"Wang","year":"2020","venue":"CVPR (workshop)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zeiler and Fergus, 2014] Matthew Zeiler and Rob Fergus. Visu- alizing and understanding convolutional networks. In European conference on computer vision, pages 818-833, 2014.","title":"Visualizing and understanding convolutional networks","authors":"Matthew Zeiler, Rob Fergus","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2021] Qinglong Zhang, Lu Rao, and Yubin Yang. A novel visual interpretability for deep neural networks by optimiz- ing activation maps with perturbation. In AAAI, volume 35 (4), pages 3377-3384, 2021.","title":"A novel visual interpretability for deep neural networks by optimizing activation maps with perturbation","authors":"Qinglong Zhang, Lu Rao, Yubin Yang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}