{"type":"ScholarlyArticle","title":"A Logic-based Approach to Contrastive Explainability for Neurosymbolic Visual Question Answering","authors":[{"name":"Thomas Eiter","slug":"thomas-eiter","given":"Thomas","family":"Eiter","affiliation":"Institute for Logic and Computation , TU Wien , Favoritenstraße 9-11 , 1040 Vienna , Austria,"},{"name":"Tobias Geibinger","slug":"tobias-geibinger","given":"Tobias","family":"Geibinger","affiliation":"Institute for Logic and Computation , TU Wien , Favoritenstraße 9-11 , 1040 Vienna , Austria,"},{"name":"Nelson Higuera","slug":"nelson-higuera","given":"Nelson","family":"Higuera","affiliation":"Institute for Logic and Computation , TU Wien , Favoritenstraße 9-11 , 1040 Vienna , Austria,"},{"name":"Johannes Oetsch","slug":"johannes-oetsch","given":"Johannes","family":"Oetsch","affiliation":"Institute for Logic and Computation , TU Wien , Favoritenstraße 9-11 , 1040 Vienna , Austria,"}],"abstract":"Visual Question Answering (VQA) is a well-known problem for which deep-learning is key. This poses a challenge for explaining answers to questions, the more if advanced notions like contrastive explanations (CEs) should be provided. The latter explain why an answer has been reached in contrast to a different one and are attractive as they focus on reasons necessary to flip a query answer. We present a CE framework for VQA that uses a neurosymbolic VQA architecture which disentangles perception from reasoning. Once the reasoning part is provided as logical theory, we use answer-set programming, in which CE generation can be framed as an abduction problem. We validate our approach on the CLEVR dataset, which we extend by more sophisticated questions to further demonstrate the robustness of the modular architecture. While we achieve top performance compared to related approaches, we can also produce CEs for explanation, model debugging, and validation tasks, showing the versatility of the declarative approach to reasoning.","doi":"10.24963/ijcai.2023/408","url":"https://ijcai.org/proceedings/2023/408","pdf":"https://ijcai.org/proceedings/2023/0408.pdf","markdown":"https://ijcai.org/proceedings/2023/408.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/408","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":408,"pages":{"first":3668,"last":3676},"keywords":["Machine Learning: ML: Explainable/Interpretable machine learning","Machine Learning: ML: Neuro-symbolic methods","Knowledge Representation and Reasoning: KRR: Logic programming"],"videos":[],"references":[{"position":1,"raw":"[Andres et al., 2012] Benjamin Andres, Benjamin Kaufmann, Oliver Matheis, and Torsten Schaub. Unsatisfiability-based optimization in clasp. In Technical Communications of the 28th International Conference on Logic Programming (ICLP 2012), volume 17 of LIPIcs, pages 211-221. Schloss Dagstuhl -Leibniz-Zentrum für Informatik, 2012.","title":"Unsatisfiability-based optimization in clasp","authors":"Benjamin Andres, Benjamin Kaufmann, Oliver Matheis, Torsten Schaub","year":"2012","venue":"Technical Communications of the 28th International Conference on Logic Programming (ICLP 2012)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arras et al., 2022] Leila Arras, Ahmed Osman, and Woj- ciech Samek. CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations. Information Fusion, 81:14-40, 2022.","title":"CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations","authors":"Leila Arras, Ahmed Osman, Wojciech Samek","year":"2022","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barra et al., 2021] Silvio Barra, Carmen Bisogni, Maria De Marsico, and Stefano Ricciardi. Visual question answering: Which investigated applications? Pattern Recognit. Lett., 151:325-331, 2021.","title":"Visual question answering: Which investigated applications?","authors":"Silvio Barra, Carmen Bisogni, Maria De Marsico, Stefano Ricciardi","year":"2021","venue":"Pattern Recognit. Lett","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beckham et al., 2023] Christopher Beckham, Martin Weiss, Florian Golemo, Sina Honari, Derek Nowrouzezahrai, and Christopher Pal. Visual question answering from another perspective: CLEVR mental rotation tests. Pattern Recog- nition, 136:109209, 2023.","title":"Visual question answering from another perspective: CLEVR mental rotation tests","authors":"Christopher Beckham, Martin Weiss, Florian Golemo, Sina Honari, Derek Nowrouzezahrai, Christopher Pal","year":"2023","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brewka et al., 2011] Gerhard Brewka, Thomas Eiter, and Miroslaw Truszczynski. Answer set programming at a glance. Commun. ACM, 54(12):92-103, 2011.","title":"Answer set programming at a glance","authors":"Gerhard Brewka, Thomas Eiter, Miroslaw Truszczynski","year":"2011","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cabalar et al., 2020] Pedro Cabalar, Jorge Fandinno, and Brais Muñiz. A system for explainable answer set pro- gramming. In Technical Communications of the 36th Inter- national Conference on Logic Programming (ICLP 2020), volume 325 of EPTCS, pages 124-136, 2020.","title":"A system for explainable answer set programming","authors":"Pedro Cabalar, Jorge Fandinno, Brais Muñiz","year":"2020","venue":"Technical Communications of the 36th International Conference on Logic Programming (ICLP 2020)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Calimeri et al., 2020] Francesco Calimeri, Wolfgang Faber, Martin Gebser, Giovambattista Ianni, Roland Kaminski, Thomas Krennwallner, Nicola Leone, Marco Maratea, Francesco Ricca, and Torsten Schaub. ASP-core-2 input language format. Theory and Practice of Logic Program- ming, 20(2):294-309, 2020.","title":"ASP-core-2 input language format","authors":"Francesco Calimeri, Wolfgang Faber, Martin Gebser, Giovambattista Ianni, Roland Kaminski, Thomas Krennwallner, Nicola Leone, Marco Maratea, Francesco Ricca, Torsten Schaub","year":"2020","venue":"Theory and Practice of Logic Programming","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosilovic et al., 2018] Filip Karlo Dosilovic, Mario Brcic, and Nikica Hlupic. Explainable artificial intelligence: A survey. In Proc. of the 41st International Convention on Information and Communication Technology, pages 210- 215. IEEE, 2018.","title":"Explainable artificial intelligence: A survey","authors":"Filip Karlo Dosilovic, Mario Brcic, Nikica Hlupic","year":"2018","venue":"Proc. of the 41st International Convention on Information and Communication Technology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Eiter et al., 2022] Thomas Eiter, Nelson Higuera, Johannes Oetsch, and Michael Pritz. A neuro-symbolic ASP pipeline for visual question answering. Theory and Practice of Logic Programming, 22(5):739-754, 2022.","title":"A neuro-symbolic ASP pipeline for visual question answering","authors":"Thomas Eiter, Nelson Higuera, Johannes Oetsch, Michael Pritz","year":"2022","venue":"Theory and Practice of Logic Programming","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Freiesleben, 2022] Timo Freiesleben. The intriguing rela- tion between counterfactual explanations and adversarial examples. Minds Mach., 32(1):77-109, 2022.","title":"The intriguing relation between counterfactual explanations and adversarial examples","authors":"Timo Freiesleben","year":"2022","venue":"Minds Mach","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gebser et al., 2019] Martin Gebser, Roland Kaminski, Ben- jamin Kaufmann, and Torsten Schaub. Multi-shot asp solv- ing with clingo. Theory and Practice of Logic Program- ming, 19(1):27-82, 2019.","title":"Multi-shot asp solving with clingo","authors":"Gebser","year":"2019","venue":"Theory and Practice of Logic Programming","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2020] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell., 42(2):386-397, 2020.","title":"Mask R-CNN","authors":"He","year":"2020","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hudson and Manning, 2018] Drew A. Hudson and Christo- pher D. Manning. Compositional attention networks for machine reasoning. In Proc. of the 6th International Con- ference on Learning Representations (ICLR 2018), 2018.","title":"Compositional attention networks for machine reasoning","authors":"Drew A Hudson, Christopher D Manning","year":"2018","venue":"Proc. of the 6th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ignatiev and Silva, 2021] Alexey Ignatiev and João P. Mar- ques Silva. Sat-based rigorous explanations for decision lists. In Proc. of the 24th International Conference Theory and Applications of Satisfiability Testing (SAT 2021), vol- ume 12831 of Lecture Notes in Computer Science, pages 251-269. Springer, 2021.","title":"Marques Silva. Sat-based rigorous explanations for decision lists","authors":"Alexey Ignatiev, P João","year":"2021","venue":"Proc. of the 24th International Conference Theory and Applications of Satisfiability Testing (SAT 2021)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ignatiev et al., 2020] Alexey Ignatiev, Nina Narodytska, Nicholas Asher, and João Marques-Silva. From contrastive to abductive explanations and back again. In In Proc. of the 19th International Conference of the Italian Association for Artificial Intelligence (AIxIA 2020), volume 12414 of Lec- ture Notes in Computer Science, pages 335-355. Springer, 2020.","title":"From contrastive to abductive explanations and back again","authors":"Alexey Ignatiev, Nina Narodytska, Nicholas Asher, João Marques-Silva","year":"2020","venue":"Proc. of the 19th International Conference of the Italian Association for Artificial Intelligence (AIxIA 2020)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jacovi et al., 2021] Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Gold- berg. Contrastive explanations for model interpretability. In Proc. of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pages 1597- 1611. Association for Computational Linguistics, 2021.","title":"Contrastive explanations for model interpretability","authors":"Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, Yoav Goldberg","year":"2021","venue":"Proc. of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proc. of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), pages 1988-1997. IEEE Computer Society, 2017.","title":"CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross B Girshick","year":"2017","venue":"Proc. of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lindstrom and Abraham, 2022] Adam Dahlgren Lindstrom and Savitha Sam Abraham. Clevr-math: A dataset for com- positional language, visual and mathematical reasoning. In Proc. of the 16th International Workshop on Neural- Symbolic Learning and Reasoning as part of the 2nd In- ternational Joint Conference on Learning & Reasoning (IJCLR 2022), volume 3212 of CEUR Workshop Proceed- ings, pages 155-170, 2022.","title":"Clevr-math: A dataset for compositional language, visual and mathematical reasoning","authors":"Abraham ; Lindstrom, Dahlgren Adam, Savitha Lindstrom, Abraham Sam","year":"2022","venue":"Proc. of the 16th International Workshop on Neural-Symbolic Learning and Reasoning as part of the 2nd International Joint Conference on Learning & Reasoning (IJCLR 2022)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lipton, 1990] Peter Lipton. Contrastive explanation. Royal Institute of Philosophy Supplement, 27:247-266, 1990.","title":"Peter Lipton. Contrastive explanation","authors":"Lipton","year":"1990","venue":"Royal Institute of Philosophy Supplement","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. In Proc. of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421. Association for Computa- tional Linguistics, 2015.","title":"Effective approaches to attentionbased neural machine translation","authors":"Minh-Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"Proc. of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Manhaeve et al., 2018] Robin Manhaeve, Sebastijan Duman- cic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic program- ming. In Proc. of Advances in Neural Information Process- ing Systems 31: Annual Conference on Neural Information Processing Systems (NeurIPS 2018), pages 3753-3763, 2018.","title":"Deepproblog: Neural probabilistic logic programming","authors":"Manhaeve","year":"2018","venue":"Proc. of Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems (NeurIPS 2018)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Marques-Silva, 2023] Joao Marques-Silva. Logic-Based Ex- plainability in Machine Learning, pages 24-104. Springer Nature Switzerland, Cham, 2023.","title":"Logic-Based Explainability in Machine Learning","authors":"Joao Marques-Silva","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Miller, 2019] Tim Miller. Explanation in artificial intelli- gence: Insights from the social sciences. Artif. Intell., 267:1-38, 2019.","title":"Explanation in artificial intelligence: Insights from the social sciences","authors":"Tim Miller, Miller","year":"2019","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Salewski et al., 2020] Leonard Salewski, A. Sophia Koepke, Hendrik P. A. Lensch, and Zeynep Akata. CLEVR-X: A visual reasoning dataset for natural language explana- tions. In xxAI -Beyond Explainable AI -International Workshop, Held in Conjunction with ICML 2020, Revised and Extended Papers, volume 13200 of Lecture Notes in Computer Science, pages 69-88. Springer, 2020.","title":"CLEVR-X: A visual reasoning dataset for natural language explanations","authors":"Leonard Salewski, A Sophia Koepke, Hendrik P A Lensch, Zeynep Akata","year":"2020","venue":"xxAI -Beyond Explainable AI -International Workshop, Held in Conjunction with ICML 2020, Revised and Extended Papers","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sampat et al., 2021] Shailaja Keyur Sampat, Akshay Kumar, Yezhou Yang, and Chitta Baral. Clevr hyp: A challenge dataset and baselines for visual question answering with hypothetical actions over images. In Proc. of the 2021 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (NAACL-HLT 2021), pages 3692-3709. Association for Computational Linguistics, 2021.","title":"Clevr hyp: A challenge dataset and baselines for visual question answering with hypothetical actions over images","authors":"Keyur Shailaja, Akshay Sampat, Yezhou Kumar, Chitta Yang, Baral","year":"2021","venue":"Proc. of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2021)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Santoro et al., 2017] Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Pe- ter W. Battaglia, and Tim Lillicrap. A simple neural net- work module for relational reasoning. In Proc. of Ad- vances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems (NeurIPS 2017), pages 4967-4976, 2017.","title":"A simple neural network module for relational reasoning","authors":"Adam Santoro, David Raposo, G T David, Mateusz Barrett, Razvan Malinowski, Peter W Pascanu, Tim Battaglia, Lillicrap","year":"2017","venue":"Proc. of Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NeurIPS 2017)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Skryagin et al., 2022] Arseny Skryagin, Wolfgang Stammer, Daniel Ochs, Devendra Singh Dhami, and Kristian Kerst- ing. Neural-Probabilistic Answer Set Programming. In Proc. of the 19th International Conference on Principles of Knowledge Representation and Reasoning (KR 2022), pages 463-473, 2022.","title":"Neural-Probabilistic Answer Set Programming","authors":"Arseny Skryagin, Wolfgang Stammer, Daniel Ochs, Devendra Singh Dhami, Kristian Kersting","year":"2022","venue":"Proc. of the 19th International Conference on Principles of Knowledge Representation and Reasoning (KR 2022)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stepin et al., 2021] Ilia Stepin, José Maria Alonso, Alejan- dro Catalá, and Martin Pereira-Fariña. A survey of con- trastive and counterfactual explanation generation meth- ods for explainable artificial intelligence. IEEE Access, 9:11974-12001, 2021.","title":"A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence","authors":"Ilia Stepin, Maria José, Alejandro Alonso, Martin Catalá, Pereira-Fariña","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Winters et al., 2022] Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt. Deepstochlog: Neural stochastic logic programming. In Proc. of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI 2022), pages 10090-10100. AAAI Press, 2022.","title":"Deepstochlog: Neural stochastic logic programming","authors":"Thomas Winters, Giuseppe Marra, Robin Manhaeve, Luc De, Raedt","year":"2022","venue":"Proc. of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI 2022)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2020] Zhun Yang, Adam Ishay, and Joohyung Lee. Neurasp: Embracing neural networks into answer set programming. In Christian Bessiere, editor, Proc. of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI 2020), pages 1755-1762, 2020.","title":"Neurasp: Embracing neural networks into answer set programming","authors":"Zhun Yang, Adam Ishay, Joohyung Lee","year":"2020","venue":"Proc. of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI 2020)","doi":null,"url":null,"internal_url":"/proceedings/2020/243"},{"position":31,"raw":"[Yi et al., 2018] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic VQA: disentangling reasoning from vision and language understanding. In Proc. of Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018), pages 1039-1050, 2018.","title":"Neural-symbolic VQA: disentangling reasoning from vision and language understanding","authors":"Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, Josh Tenenbaum","year":"2018","venue":"Proc. of Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zou and Xie, 2020] Yeyun Zou and Qiyu Xie. A survey on VQA: Datasets and approaches. In Proc. of the 2nd In- ternational Conference on Information Technology and Computer Application (ITCA 2020), pages 289-297, 2020.","title":"A survey on VQA: Datasets and approaches","authors":"Xie Zou, Yeyun Zou, Qiyu Xie","year":"2020","venue":"Proc. of the 2nd International Conference on Information Technology and Computer Application (ITCA 2020)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}