{"type":"ScholarlyArticle","title":"Twin-Systems to Explain Artificial Neural Networks using Case-Based Reasoning: Comparative Tests of Feature-Weighting Methods in ANN-CBR Twins for XAI","authors":[{"name":"Eoin M. Kenny","slug":"eoin-m-kenny","given":"Eoin M.","family":"Kenny","affiliation":"School of Computer Science , University College Dublin , Dublin , Ireland; Insight Centre for Data Analytics , Dublin , Ireland"},{"name":"Mark T. Keane","slug":"mark-t-keane","given":"Mark T.","family":"Keane","affiliation":"School of Computer Science , University College Dublin , Dublin , Ireland; Insight Centre for Data Analytics , Dublin , Ireland; VistaMilk SFI Research Centre , Dublin , Ireland"}],"abstract":"In this paper, twin-systems are described to address the eXplainable artificial intelligence (XAI) problem, where a black box model is mapped to a white box “twin” that is more interpretable, with both systems using the same dataset. The framework is instantiated by twinning an artificial neural network (ANN; black box) with a case-based reasoning system (CBR; white box), and mapping the feature weights from the former to the latter to find cases that explain the ANN’s outputs. Using a novel evaluation method, the effectiveness of this twin-system approach is demonstrated by showing that nearest neighbor cases can be found to match the ANN predictions for benchmark datasets. Several feature-weighting methods are competitively tested in two experiments, including our novel, contributions-based method (called COLE) that is found to perform best. The tests consider the ”twinning” of traditional multilayer perceptron (MLP) networks and convolutional neural networks (CNN) with CBR systems. For the CNNs trained on image data, qualitative evidence shows that cases provide plausible explanations for the CNN’s classifications.","doi":"10.24963/ijcai.2019/376","url":"https://ijcai.org/proceedings/2019/376","pdf":"https://ijcai.org/proceedings/2019/0376.pdf","markdown":"https://ijcai.org/proceedings/2019/376.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/376","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":376,"pages":{"first":2708,"last":2715},"keywords":["Machine Learning: Interpretability","Knowledge Representation and Reasoning: Case-based reasoning","Machine Learning: Deep Learning","Machine Learning: Explainable Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Amin et al., 2018] Kareem Amin, Stelios Kapetanakis, Klaus-Dieter Althoff, Andreas Dengel, and Miltos Petridis. Answering with cases: A CBR approach to deep learning. In International Conference on Case-Based Rea- soning, pages 15-27. Springer, 2018.","title":"Answering with cases: A CBR approach to deep learning","authors":"Kareem Amin, Stelios Kapetanakis, Klaus-Dieter Althoff, Andreas Dengel, Miltos Petridis","year":"2018","venue":"International Conference on Case-Based Reasoning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bach et al., 2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.","title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","authors":"Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, Wojciech Samek","year":"2015","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2011] Runbo Bai, Hailei Jia, and Pingzhou Cao. Factor Sensitivity Analysis with Neural Network Simula- tion based on Perturbation System. Journal of Computers, 6(7), July 2011.","title":"Factor Sensitivity Analysis with Neural Network Simulation based on Perturbation System","authors":"Bai","year":"2011","venue":"Journal of Computers","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bench-Capon, 2017] Trevor JM Bench-Capon. Hypo's legacy: introduction to the virtual special issue. Artificial Intelligence and Law, 25(2):205-250, 2017.","title":"Hypo's legacy: introduction to the virtual special issue","authors":"J M Trevor, Bench-Capon","year":"2017","venue":"Artificial Intelligence and Law","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018] Chaofan Chen, Oscar Li, Alina Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. arXiv preprint arXiv:1806.10574, 2018.","title":"This looks like that: deep learning for interpretable image recognition","authors":"Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clancey, 1983] William J. Clancey. The epistemology of a rule-based expert system -a framework for explanation. Artificial Intelligence, 20(3):215-251, May 1983.","title":"The epistemology of a rule-based expert system -a framework for explanation","authors":"William J Clancey","year":"1983","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cunningham et al., 2003] Pádraig Cunningham, Dónal Doyle, and John Loughrey. An Evaluation of the Use- fulness of Case-Based Explanation. In Case-Based Reasoning Research and Development, volume 2689, pages 122-130. Springer Berlin Heidelberg, Berlin, Heidelberg, 2003.","title":"An Evaluation of the Usefulness of Case-Based Explanation","authors":"Cunningham","year":"2003","venue":"Case-Based Reasoning Research and Development","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[De Mantaras et al., 2005] Ramon Lopez De Mantaras, David McSherry, Derek Bridge, David Leake, Barry Smyth, Susan Craw, Boi Faltings, Mary Lou Maher, Michael T Cox, Kenneth Forbus, et al. Retrieval, reuse, revision and retention in case-based reasoning. The Knowledge Engineering Review, 20(3):215-240, 2005.","title":"Retrieval, reuse, revision and retention in case-based reasoning","authors":"Ramon Lopez, De Mantaras, David Mcsherry, Derek Bridge, David Leake, Barry Smyth, Susan Craw, Boi Faltings, Mary Lou Maher, Michael T Cox, Kenneth Forbus","year":"2005","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[de Oña and Garrido, 2014] Juan de Oña and Concepción Garrido. Extracting the contribution of independent vari- ables in neural network models: a new approach to han- dle instability. Neural Computing and Applications, 25(3- 4):859-869, September 2014.","title":"Juan de Oña and Concepción Garrido. Extracting the contribution of independent variables in neural network models: a new approach to handle instability","authors":"De Oña, Garrido","year":"2014","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Erhan et al., 2009] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pascal Vincent, and P O Box. Visualizing Higher-Layer Features of a Deep Network. page 14, 2009.","title":"Visualizing Higher-Layer Features of a Deep Network","authors":"Yoshua Dumitru Erhan, Aaron Bengio, Pascal Courville, P O Vincent, Box","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garson, 1991] G David Garson. Interpreting neural- network connection weights. AI expert, 6(4):46-51, 1991.","title":"Interpreting neuralnetwork connection weights","authors":"Garson David","year":"1991","venue":"AI expert","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Keane and Kenny, 2019a] Mark T. Keane and Eoin M. Kenny. How case-based reasoning explains neural net- works: A theoretical analysis of XAI using post-hoc explanation-by-example from a survey of ANN-CBR twin-systems. arXiv preprint arXiv:1905.07186, 2019a.","title":"How case-based reasoning explains neural networks: A theoretical analysis of XAI using post-hoc explanation-by-example from a survey of ANN-CBR twin-systems","authors":"Kenny ; A] Keane, T Mark, Eoin M Keane, Kenny","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Keane and Kenny, 2019b] Mark T. Keane and Eoin M. Kenny. The twin-system approach as one generic solution for XAI: An overview of ANN-CBR twins for explaining deep learning. arXiv preprint arXiv:1905.08069, 2019b.","title":"The twin-system approach as one generic solution for XAI: An overview of ANN-CBR twins for explaining deep learning","authors":"Kenny ; B] Keane, T Mark, Eoin M Keane, Kenny","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017] Oscar Li, Hao Liu, Chaofan Chen, and Cyn- thia Rudin. Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions. arXiv:1710.04806 [cs, stat], October 2017. arXiv: 1710.04806.","title":"Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions","authors":"Oscar Li, Hao Liu, Chaofan Chen, Cynthia Rudin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lipton, 2018] Zach C. Lipton. The mythos of model inter- pretability. Queue, 16(3):30, 2018.","title":"The mythos of model interpretability","authors":"Zach C Lipton","year":"2018","venue":"Queue","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lundberg and Lee, 2017] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Medsker, 2012] Larry R Medsker. Hybrid intelligent sys- tems. Springer Science & Business Media, 2012.","title":"Hybrid intelligent systems","authors":"Larry R Medsker","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nugent and Cunningham, 2005] Conor Nugent and Pádraig Cunningham. A Case-Based Explanation System for Black-Box Systems. Artificial Intelligence Review, 24(2):163-178, October 2005.","title":"Conor Nugent and Pádraig Cunningham. A Case-Based Explanation System for Black-Box Systems","authors":"Cunningham Nugent","year":"2005","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Olden and Jackson, 2002] Julian D Olden and Donald A Jackson. Illuminating the \"black box\": a randomization approach for understanding variable contributions in ar- tificial neural networks. Ecological Modelling, 154(1- 2):135-150, August 2002.","title":"Illuminating the \"black box\": a randomization approach for understanding variable contributions in artificial neural networks","authors":"D Julian, Donald A Olden, Jackson","year":"2002","venue":"Ecological Modelling","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Olden et al., 2004] Julian D Olden, Michael K Joy, and Russell G Death. An accurate comparison of methods for quantifying variable importance in artificial neural net- works using simulated data. Ecological Modelling, 178(3- 4):389-397, November 2004.","title":"An accurate comparison of methods for quantifying variable importance in artificial neural networks using simulated data","authors":"Michael K Julian D Olden, Russell G Joy, Death","year":"2004","venue":"Ecological Modelling","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Papernot and McDaniel, 2018] Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018.","title":"Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning","authors":"Nicolas Papernot, Patrick Mcdaniel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Park et al., 2004] Jae Heon Park, Kwang Hyuk Im, Chung- Kwan Shin, and Sang Chan Park. MBNR: Case-Based Reasoning with Local Feature Weighting by Neural Net- work. Applied Intelligence, 21(3):265-276, November 2004.","title":"MBNR: Case-Based Reasoning with Local Feature Weighting by Neural Network","authors":"Jae Heon, Park, Kwang Hyuk Im, Chung-Kwan Shin, Sang Chan Park","year":"2004","venue":"Applied Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pei et al., 2017] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In proceedings of the 26th Sympo- sium on Operating Systems Principles, pages 1-18. ACM, 2017.","title":"Deepxplore: Automated whitebox testing of deep learning systems","authors":"Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana","year":"2017","venue":"proceedings of the 26th Symposium on Operating Systems Principles","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reategui et al., 1997] E.B Reategui, J.A Campbell, and B.F Leao. Combining a neural network with case-based rea- soning in a diagnostic system. Artificial Intelligence in Medicine, 9(1):5-27, January 1997.","title":"Combining a neural network with case-based reasoning in a diagnostic system","authors":"E Reategui, J A Campbell, B F Leao","year":"1997","venue":"Artificial Intelligence in Medicine","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowl- edge discovery and data mining, pages 1135-1144. ACM, 2016.","title":"Why should i trust you?: Explaining the predictions of any classifier","authors":"Ribeiro","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sani et al., 2017] Sadiq Sani, Nirmalie Wiratunga, and Stewart Massie. Learning deep features for k-NN-based human activity recognition. In Proceedings of the ICCBR- 17 Workshop. ICCBR (Organisers), 2017.","title":"Learning deep features for k-NN-based human activity recognition","authors":"Sadiq Sani, Nirmalie Wiratunga, Stewart Massie","year":"2017","venue":"Proceedings of the ICCBR-17 Workshop. ICCBR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explana- tions from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shin et al., 2000] Chung-Kwan Shin, Ui Tak Yun, Huy Kang Kim, and Sang Chan Park. A hybrid ap- proach of neural network and memory-based learning to data mining. IEEE Transactions on Neural Networks, 11(3):637-646, 2000.","title":"A hybrid approach of neural network and memory-based learning to data mining","authors":"Chung-Kwan Shin, Ui Tak Yun, Huy Kang Kim, Sang Chan Park","year":"2000","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shrikumar et al., 2017] Avanti Shrikumar, Peyton Green- side, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceed- ings of the 34th International Conference on Machine Learning-Volume 70, pages 3145-3153. JMLR. org, 2017.","title":"Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences","authors":"Shrikumar","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sundararajan et al., 2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep net- works. In Proceedings of the 34th International Confer- ence on Machine Learning-Volume 70, pages 3319-3328. JMLR. org, 2017.","title":"Axiomatic attribution for deep networks","authors":"Mukund Sundararajan, Ankur Taly, Qiqi Yan","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sørmo et al., 2005] Frode Sørmo, Jörg Cassens, and Ag- nar Aamodt. Explanation in Case-Based Reason- ing:Perspectives and goals. Artificial Intelligence Review, 24(2):109-143, October 2005.","title":"Explanation in Case-Based Reasoning:Perspectives and goals","authors":"Frode Sørmo, Jörg Cassens, Agnar Aamodt","year":"2005","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tintarev and Masthoff, 2007] Nava Tintarev and Judith Masthoff. A survey of explanations in recommender sys- tems. In 2007 IEEE 23rd International Conference on Data Engineering Workshop, pages 801-810, Istanbul, Turkey, April 2007. IEEE.","title":"A survey of explanations in recommender systems","authors":"Nava Tintarev, Judith Masthoff","year":"2007","venue":"IEEE 23rd International Conference on Data Engineering Workshop","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2018] Chen-Lin Zhang, Jian-Hao Luo, Xiu- Shen Wei, and Jianxin Wu. In defense of fully connected layers in visual representation transfer. In Advances in Multimedia Information Processing -PCM 2017, volume 10736, pages 807-817. Springer International Publishing, Cham, 2018.","title":"In defense of fully connected layers in visual representation transfer","authors":"Chen-Lin Zhang, Jian-Hao Luo, Xiu-Shen Wei, Jianxin Wu","year":"2018","venue":"Advances in Multimedia Information Processing -PCM 2017","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2015] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object de- tectors emerge in deep scene cnns. 2015.","title":"Object detectors emerge in deep scene cnns","authors":"Zhou","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}