{"type":"ScholarlyArticle","title":"Generating Natural Counterfactual Visual Explanations","authors":[{"name":"Wenqi Zhao","slug":"wenqi-zhao","given":"Wenqi","family":"Zhao","affiliation":"Graduate School of Information Science and Technology , Hokkaido University"},{"name":"Satoshi Oyama","slug":"satoshi-oyama","given":"Satoshi","family":"Oyama","affiliation":"Graduate School of Information Science and Technology , Hokkaido University; Global Institution for Collaborative Research and Education , Hokkaido University"},{"name":"Masahito Kurihara","slug":"masahito-kurihara","given":"Masahito","family":"Kurihara","affiliation":"Graduate School of Information Science and Technology , Hokkaido University"}],"abstract":"Counterfactual explanations help users to understand the behaviors of machine learning models by changing the inputs for the existing outputs. For an image classification task, an example counterfactual visual explanation explains: \"for an example that belongs to class A, what changes do we need to make to the input so that the output is more inclined to class B.\" Our research considers changing the attribute description text of class A on the basis of the attributes of class B and generating counterfactual images on the basis of the modified text. We can use the prediction results of the model on counterfactual images to find the attributes that have the greatest effect when the model is predicting classes A and B. We applied our method to a fine-grained image classification dataset and used the generative adversarial network to generate natural counterfactual visual explanations. To evaluate these explanations, we used them to assist crowdsourcing workers in an image classification task. We found that, within a specific range, they improved classification accuracy.","doi":"10.24963/ijcai.2020/742","url":"https://ijcai.org/proceedings/2020/742","pdf":"https://ijcai.org/proceedings/2020/0742.pdf","markdown":"https://ijcai.org/proceedings/2020/742.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/742","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":742,"pages":{"first":5204,"last":5205},"keywords":["Machine Learning: Interpretability","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25472/file_25472.m3u8","length_sec":274,"provider":"hls"}],"references":[{"position":1,"raw":"[Doshi-Velez and Kim, 2017] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.","title":"Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning","authors":"Kim Doshi-Velez","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Goyal et al., 2019] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. arXiv preprint arXiv:1904.07451, 2019.","title":"Counterfactual visual explanations","authors":"Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, Stefan Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hendricks et al., 2018] Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Generating counterfactual explanations with natural language. arXiv preprint arXiv:1806.09809, 2018.","title":"Generating counterfactual explanations with natural language","authors":"Anne Lisa, Ronghang Hendricks, Trevor Hu, Zeynep Darrell, Akata","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Xu et al., 2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xi- aodong He. Attngan: Fine-grained text to image gener- ation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316-1324, 2018.","title":"Attngan: Fine-grained text to image generation with attentional generative adversarial networks","authors":"Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Zhang et al., 2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial net- works. In Proceedings of the IEEE international confer- ence on computer vision, pages 5907-5915, 2017.","title":"Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks","authors":"Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris N Metaxas","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}