{"type":"ScholarlyArticle","title":"ExplainIt!: A Tool for Computing Robust Attributions of DNNs","authors":[{"name":"Sumit Jha","slug":"sumit-jha","given":"Sumit","family":"Jha","affiliation":"Computer Science Department , University of Texas at San Antonio , San Antonio , TX 78249"},{"name":"Alvaro Velasquez","slug":"alvaro-velasquez","given":"Alvaro","family":"Velasquez","affiliation":"Information Directorate , Air Force Research Laboratory , Rome , NY13441"},{"name":"Rickard Ewetz","slug":"rickard-ewetz","given":"Rickard","family":"Ewetz","affiliation":"ECE Department , University of Central Florida , Orlando , FL 32816"},{"name":"Laura Pullum","slug":"laura-pullum","given":"Laura","family":"Pullum","affiliation":"Computer Science and Maths Division , Oak Ridge National Laboratory , Oak Ridge , TN 37830"},{"name":"Susmit Jha","slug":"susmit-jha","given":"Susmit","family":"Jha","affiliation":"Computer Science Laboratory , SRI International , Menlo Park , CA 94709"}],"abstract":"Responsible integration of deep neural networks into the design of trustworthy systems requires the ability to explain decisions made by these models. Explainability and transparency are critical for system analysis, certification, and human-machine teaming. We have recently demonstrated that neural stochastic differential equations (SDEs) present an explanation-friendly DNN  architecture. In this paper, we present ExplainIt, an online tool for explaining AI decisions that uses neural SDEs to create visually sharper and more robust attributions than traditional residual neural networks. Our tool shows that the injection of noise in every layer of a residual network often leads to less noisy and less fragile integrated gradient attributions. The discrete neural stochastic differential equation model is trained on the ImageNet data set with a million images, and the demonstration produces robust attributions on images in the ImageNet validation library and on a variety of images in the wild. Our online tool is hosted publicly for educational purposes.","doi":"10.24963/ijcai.2022/853","url":"https://ijcai.org/proceedings/2022/853","pdf":"https://ijcai.org/proceedings/2022/0853.pdf","markdown":"https://ijcai.org/proceedings/2022/853.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/853","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Demo Track","track":"Demo Track","number":853,"pages":{"first":5916,"last":5919},"keywords":["AI Ethics, Trust, Fairness: Explainability and Interpretability"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987215","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985358","length_sec":358,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2015] Yunjin Chen, Wei Yu, and Thomas Pock. On learning optimized reaction diffusion processes for effective image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5261-5269, 2015.","title":"On learning optimized reaction diffusion processes for effective image restoration","authors":"Yunjin Chen, Wei Yu, Thomas Pock","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Core et al., 2006] Mark G Core, H Chad Lane, Michael Van Lent, Dave Gomboc, Steve Solomon, Milton Rosenberg, et al. Building explainable artificial intelligence systems. In AAAI, pages 1766-1773, 2006.","title":"Building explainable artificial intelligence systems","authors":"Chad Mark G Core, Michael Lane, Dave Van Lent, Steve Gomboc, Milton Solomon, Rosenberg","year":"2006","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cramer et al., 2022] Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Nonlinear isometric manifold learning for injective normalizing flows. arXiv preprint arXiv:2203.03934, 2022.","title":"Nonlinear isometric manifold learning for injective normalizing flows","authors":"Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, Manuel Dahmen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghorbani et al., 2019] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681-3688, 2019.","title":"Interpretation of neural networks is fragile","authors":"Ghorbani","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hammond, 1990] Kristian J Hammond. Explaining and repairing plans that fail. Artificial intelligence, 45(1-2):173- 228, 1990.","title":"Explaining and repairing plans that fail","authors":"Kristian J Hammond","year":"1990","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jha and Raman, 2016] Susmit Jha and Vasumathi Raman. Automated synthesis of safe autonomous vehicle control under perception uncertainty. In NASA Formal Methods Symposium, pages 117-132. Springer, 2016.","title":"Susmit Jha and Vasumathi Raman. Automated synthesis of safe autonomous vehicle control under perception uncertainty","authors":"Raman Jha","year":"2016","venue":"NASA Formal Methods Symposium","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jha et al., 2017] Susmit Jha, Vasumathi Raman, Alessandro Pinto, Tuhin Sahai, and Michael Francis. On learning sparse Boolean formulae for explaining AI decisions. In NASA Formal Methods Symposium, pages 99-114. Springer, 2017.","title":"On learning sparse Boolean formulae for explaining AI decisions","authors":"Susmit Jha, Vasumathi Raman, Alessandro Pinto, Tuhin Sahai, Michael Francis","year":"2017","venue":"NASA Formal Methods Symposium","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jha et al., 2019a] Susmit Jha, Sunny Raj, Steven Fernandes, Sumit K Jha, Somesh Jha, Brian Jalaian, Gunjan Verma, and Ananthram Swami. Attribution-based confidence metric for deep neural networks. In Advances in Neural Information Processing Systems, pages 11826-11837, 2019.","title":"Attribution-based confidence metric for deep neural networks","authors":"Jha","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jha et al., 2019b] Susmit Jha, Tuhin Sahai, Vasumathi Raman, Alessandro Pinto, and Michael Francis. Explaining AI decisions using efficient methods for learning sparse Boolean formulae. Journal of Automated Reasoning, 63(4):1055-1075, 2019.","title":"Explaining AI decisions using efficient methods for learning sparse Boolean formulae","authors":"Jha","year":"2019","venue":"Journal of Automated Reasoning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jha et al., 2021] Sumit Jha, Rickard Ewetz, Alvaro Velasquez, and Susmit Jha. On smoother attributions using neural stochastic differential equations. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 522-528, 2021.","title":"On smoother attributions using neural stochastic differential equations","authors":"Sumit Jha, Rickard Ewetz, Alvaro Velasquez, Susmit Jha","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21","doi":null,"url":null,"internal_url":"/proceedings/2021/73"},{"position":12,"raw":"[Jha et al., 2022] Sumit Jha, Rickard Ewetz, Alvaro Velasquez, Arvind Ramanathan, and Susmit Jha. Shaping noise for robust attributions in neural stochastic differential equations. 36th AAAI Conference on Artificial Intelligence (AAAI), 2022.","title":"Shaping noise for robust attributions in neural stochastic differential equations","authors":"Sumit Jha, Rickard Ewetz, Alvaro Velasquez, Arvind Ramanathan, Susmit Jha","year":"2022","venue":"36th AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lane et al., 2005] H Chad Lane, Mark G Core, Michael Van Lent, Steve Solomon, and Dave Gomboc. Explainable artificial intelligence for training and tutoring. Technical report, University of Southern California, 2005.","title":"Explainable artificial intelligence for training and tutoring","authors":"Chad Lane, Michael Mark G Core, Steve Van Lent, Dave Solomon, Gomboc","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Yu, 2015] Guanbin Li and Yizhou Yu. Visual saliency based on multiscale deep features. In CVPR, pages 5455- 5463, 2015.","title":"Visual saliency based on multiscale deep features","authors":"Yu ; ] Guanbin Li, Yizhou Li, Yu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369-385, 2018.","title":"Towards robust neural networks via random self-ensemble","authors":"Liu","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2020] Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. How does noise help robustness? explanation and exploration under the neural sde framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.","title":"How does noise help robustness? explanation and exploration under the neural sde framework","authors":"Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, Cho-Jui Hsieh","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2018] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pages 3276-3285. PMLR, 2018.","title":"Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations","authors":"Yiping Lu, Aoxiao Zhong, Quanzheng Li, Bin Dong","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lundberg and Lee, 2017] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems, pages 4765-4774, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simmons, 1988] Reid G Simmons. A theory of debugging plans and interpretations. In AAAI, pages 94-99, 1988.","title":"A theory of debugging plans and interpretations","authors":"G Reid, Simmons","year":"1988","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sonoda and Murata, 2017] Sho Sonoda and Noboru Murata. Double continuum limit of deep neural networks. In ICML Workshop Principled Approaches to Deep Learning, volume 1740, 2017.","title":"Double continuum limit of deep neural networks","authors":"Murata ; ] Sho Sonoda, Noboru Sonoda, Murata","year":"2017","venue":"ICML Workshop Principled Approaches to Deep Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sundararajan et al., 2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, pages 3319-3328. JMLR. org, 2017.","title":"Axiomatic attribution for deep networks","authors":"Mukund Sundararajan, Ankur Taly, Qiqi Yan","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Swartout, 1983] William R Swartout. Xplain: A system for creating and explaining expert consulting programs. Artificial intelligence, 21(3):285-325, 1983.","title":"Xplain: A system for creating and explaining expert consulting programs","authors":"William R Swartout","year":"1983","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] Bao Wang, Zuoqiang Shi, and Stanley Osher. Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies. In Advances in Neural Information Processing Systems, volume 32, pages 1657-1667, 2019.","title":"Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies","authors":"Bao Wang, Zuoqiang Shi, Stanley Osher","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weinan, 2017] E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.","title":"A proposal on machine learning via dynamical systems","authors":"Weinan","year":"2017","venue":"Communications in Mathematics and Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yeh et al., 2019] Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32:10967-10978, 2019.","title":"On the (in) fidelity and sensitivity of explanations","authors":"Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I Inouye, Pradeep K Ravikumar","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yi et al., 2016] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In ECCV, pages 467-483. Springer, 2016.","title":"Lift: Learned invariant feature transform","authors":"Kwang Moo, Yi, Eduard Trulls, Vincent Lepetit, Pascal Fua","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}