{"type":"ScholarlyArticle","title":"TFCD: Towards Multi-modal Sarcasm Detection via Training-Free Counterfactual Debiasing","authors":[{"name":"Zhihong Zhu","slug":"zhihong-zhu","given":"Zhihong","family":"Zhu","affiliation":"Peking University"},{"name":"Xianwei Zhuang","slug":"xianwei-zhuang","given":"Xianwei","family":"Zhuang","affiliation":"Peking University"},{"name":"Yunyan Zhang","slug":"yunyan-zhang","given":"Yunyan","family":"Zhang","affiliation":"Jarvis Research Center , Tencent YouTu Lab"},{"name":"Derong Xu","slug":"derong-xu","given":"Derong","family":"Xu","affiliation":"University of Science and Technology of China"},{"name":"Guimin Hu","slug":"guimin-hu","given":"Guimin","family":"Hu","affiliation":"University of Copenhagen"},{"name":"Xian Wu","slug":"xian-wu","given":"Xian","family":"Wu","affiliation":"Jarvis Research Center , Tencent YouTu Lab"},{"name":"Yefeng Zheng","slug":"yefeng-zheng","given":"Yefeng","family":"Zheng","affiliation":"Jarvis Research Center , Tencent YouTu Lab"}],"abstract":"Multi-modal sarcasm detection (MSD), which aims to identify whether a given sample with multi-modal information (i.e., text and image) is sarcastic, has garnered widespread attention. Recent approaches focus on designing sophisticated architectures or mechanisms to extract sarcastic cues from entire or local image and text features. Nevertheless, a long-overlooked issue is that current MSD task invariably suffers from unintended dataset biases, especially the statistical label bias and sarcasmless word bias. Concretely, such harmful biases are confounders that may mislead existing models to learn spurious correlations, significantly limiting models' performance. To tackle this issue, this paper proposes a Training-Free Counterfactual Debiasing framework TFCD, which first formulates the causalities among variables in MSD via a tailored causal graph. Then, TFCD extracts biases from the conventionally-trained model by generating counterfactual utterances and contexts and mitigates them using element-wise subtraction. Extensive experiments on two benchmarks demonstrate the effectiveness of the proposed TFCD. Remarkably, TFCD requires neither data balancing nor model modifications, and thus can be seamlessly integrated into diverse state-of-the-art approaches and achieve considerable improvement margins.","doi":"10.24963/ijcai.2024/739","url":"https://ijcai.org/proceedings/2024/739","pdf":"https://ijcai.org/proceedings/2024/0739.pdf","markdown":"https://ijcai.org/proceedings/2024/739.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/739","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":739,"pages":{"first":6687,"last":6695},"keywords":["Natural Language Processing: NLP: Sentiment analysis, stylistic analysis, and argument mining"],"videos":[],"references":[{"position":1,"raw":"[Cai et al., 2019] Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sarcasm detection in Twitter with hierarchi- cal fusion model. In ACL, 2019.","title":"Multi-modal sarcasm detection in Twitter with hierarchical fusion model","authors":"Cai","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2023] Ziwei Chen, Linmei Hu, Weixin Li, Yingxia Shao, and Liqiang Nie. Causal intervention and counterfactual reasoning for multi-modal fake news detec- tion. In ACL, 2023.","title":"Causal intervention and counterfactual reasoning for multi-modal fake news detection","authors":"Ziwei Chen, Linmei Hu, Weixin Li, Yingxia Shao, Liqiang Nie","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2024a] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. HALC: Object hallucination reduction via adaptive focal-contrast decod- ing. ICML, 2024.","title":"HALC: Object hallucination reduction via adaptive focal-contrast decoding","authors":"Chen","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2024b] Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, and Huaxiu Yao. AutoPRM: Automating procedural supervision for multi-step reasoning via controllable question decomposi- tion. NAACL, 2024.","title":"AutoPRM: Automating procedural supervision for multi-step reasoning via controllable question decomposition","authors":"Chen","year":"2024","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jia et al., 2024] Mengzhao Jia, Can Xie, and Liqiang Jing. Debiasing multimodal sarcasm detection with contrastive learning. In AAAI, 2024.","title":"Debiasing multimodal sarcasm detection with contrastive learning","authors":"Jia","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kang et al., 2019] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classi- fier for long-tailed recognition. arXiv, 2019.","title":"Decoupling representation and classifier for long-tailed recognition","authors":"Kang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2023] Yong Li, Yuanzhi Wang, and Zhen Cui. De- coupled multimodal distilling for emotion recognition. In CVPR, 2023.","title":"Decoupled multimodal distilling for emotion recognition","authors":"Yong Li, Yuanzhi Wang, Zhen Cui","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liang et al., 2021] Bin Liang, Chenwei Lou, Xiang Li, Lin Gui, Min Yang, and Ruifeng Xu. Multi-modal sarcasm de- tection with interactive in-modal and cross-modal graphs. In ACM MM, 2021.","title":"Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs","authors":"Bin Liang, Chenwei Lou, Xiang Li, Lin Gui, Min Yang, Ruifeng Xu","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liang et al., 2022] Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. Multi-modal sarcasm detection via cross-modal graph convolutional network. In ACL, 2022.","title":"Multi-modal sarcasm detection via cross-modal graph convolutional network","authors":"Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, Ruifeng Xu","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2022] Zhiyu Lin, Yifei Gao, and Jitao Sang. In- vestigating and explaining the frequency bias in image classification. In IJCAI, 2022.","title":"Investigating and explaining the frequency bias in image classification","authors":"Zhiyu Lin, Yifei Gao, Jitao Sang","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/101"},{"position":11,"raw":"[Liu et al., 2022] Hui Liu, Wenya Wang, and Haoliang Li. Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement. In EMNLP, 2022.","title":"Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement","authors":"Hui Liu, Wenya Wang, Haoliang Li","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Niu et al., 2021] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Coun- terfactual VQA: A cause-effect look at language bias. In CVPR, 2021.","title":"Counterfactual VQA: A cause-effect look at language bias","authors":"Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, Ji-Rong Wen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2020] Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In EMNLP Findings, 2020.","title":"Modeling intra and inter-modality incongruity for multi-modal sarcasm detection","authors":"Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, Weiping Wang","year":"2020","venue":"EMNLP Findings","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pan et al., 2023] Hang Pan, Jiawei Chen, Fuli Feng, Wen- tao Shi, Junkang Wu, and Xiangnan He. Discriminative- invariant representation learning for unbiased recommen- dation. In IJCAI, 2023.","title":"Discriminativeinvariant representation learning for unbiased recommendation","authors":"Pan","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurIPS, 2017.","title":"Automatic differentiation in PyTorch","authors":"Paszke","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pearl and Mackenzie, 2018] Judea Pearl and Dana Macken- zie. The book of why: the new science of cause and effect. Basic Books, 2018.","title":"Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect","authors":"Mackenzie Pearl","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pearl et al., 2016] Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.","title":"Causal inference in statistics: A primer","authors":"Pearl","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Poria et al., 2016] Soujanya Poria, Erik Cambria, Deva- manyu Hazarika, and Prateek Vij. A deeper look into sar- castic tweets using deep convolutional neural networks. In COLING, 2016.","title":"A deeper look into sarcastic tweets using deep convolutional neural networks","authors":"Poria","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qian et al., 2020] Chen Qian, Fuli Feng, Lijie Wen, Li Lin, and Tat-Seng Chua. Enhancing text classification via dis- covering additional semantic clues from logograms. In SI- GIR, 2020.","title":"Enhancing text classification via discovering additional semantic clues from logograms","authors":"Chen Qian, Fuli Feng, Lijie Wen, Li Lin, Tat-Seng Chua","year":"2020","venue":"SI-GIR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qian et al., 2021] Chen Qian, Fuli Feng, Lijie Wen, Chun- ping Ma, and Pengjun Xie. Counterfactual inference for text classification debiasing. In ACL, 2021.","title":"Counterfactual inference for text classification debiasing","authors":"Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, Pengjun Xie","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qin et al., 2023] Libo Qin, Shijue Huang, Qiguang Chen, Chenran Cai, Yudi Zhang, Bin Liang, Wanxiang Che, and Ruifeng Xu. MMSD2.0: Towards a reliable multi-modal sarcasm detection system. In ACL Findings, 2023.","title":"MMSD2.0: Towards a reliable multi-modal sarcasm detection system","authors":"Libo Qin, Shijue Huang, Qiguang Chen, Chenran Cai, Yudi Zhang, Bin Liang, Wanxiang Che, Ruifeng Xu","year":"2023","venue":"ACL Findings","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In ICML, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Riloff et al., 2013] Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, 2013.","title":"Sarcasm as contrast between a positive sentiment and negative situation","authors":"Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, Ruihong Huang","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schifanella et al., 2016] Rossano Schifanella, Paloma de Juan, Joel R. Tetreault, and Liangliang Cao. Detecting sarcasm in multimodal social platforms. In ACM MM, 2016.","title":"Detecting sarcasm in multimodal social platforms","authors":"Rossano Schifanella, Paloma De Juan, Joel R Tetreault, Liangliang Cao","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2022] Teng Sun, Wenjie Wang, Liqiang Jing, Yi- ran Cui, Xuemeng Song, and Liqiang Nie. Counterfac- tual reasoning for out-of-distribution multimodal senti- ment analysis. In ACM MM, 2022.","title":"Counterfactual reasoning for out-of-distribution multimodal sentiment analysis","authors":"Teng Sun, Wenjie Wang, Liqiang Jing, Yiran Cui, Xuemeng Song, Liqiang Nie","year":"2022","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal Transformer for un- aligned multimodal language sequences. In ACL, 2019.","title":"Multimodal Transformer for unaligned multimodal language sequences","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tu et al., 2023] Geng Tu, Ran Jing, Bin Liang, Min Yang, Kam-Fai Wong, and Ruifeng Xu. A training-free debias- ing framework with counterfactual reasoning for conver- sational emotion detection. In EMNLP, 2023.","title":"A training-free debiasing framework with counterfactual reasoning for conversational emotion detection","authors":"Geng Tu, Ran Jing, Bin Liang, Min Yang, Kam-Fai Wong, Ruifeng Xu","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Culotta, 2021] Zhao Wang and Aron Culotta. Robustness to spurious correlations in text classification via automatically generated counterfactuals. In AAAI, 2021.","title":"Robustness to spurious correlations in text classification via automatically generated counterfactuals","authors":"Zhao Wang, Aron Culotta","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xin et al., 2023] Yifei Xin, Dongchao Yang, Fan Cui, Yujun Wang, and Yuexian Zou. Improving weakly supervised sound event detection with causal intervention. In ICASSP, 2023.","title":"Improving weakly supervised sound event detection with causal intervention","authors":"Yifei Xin, Dongchao Yang, Fan Cui, Yujun Wang, Yuexian Zou","year":"2023","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2020] Nan Xu, Zhixiong Zeng, and Wenji Mao. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In ACL, 2020.","title":"Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association","authors":"Nan Xu, Zhixiong Zeng, Wenji Mao","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2024a] Dingkang Yang, Mingcheng Li, Dongling Xiao, Yang Liu, Kun Yang, Zhaoyu Chen, Yuzheng Wang, Peng Zhai, Ke Li, and Lihua Zhang. Towards multimodal sentiment analysis debiasing via bias purification. arXiv, 2024.","title":"Towards multimodal sentiment analysis debiasing via bias purification","authors":"Yang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2024b] Dingkang Yang, Kun Yang, Mingcheng Li, Shunli Wang, Shuaibing Wang, and Lihua Zhang. Ro- bust emotion recognition in context debiasing. CVPR, 2024.","title":"Robust emotion recognition in context debiasing","authors":"Yang","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2016] Meishan Zhang, Yue Zhang, and Guo- hong Fu. Tweet sarcasm detection using deep neural net- work. In COLING, 2016.","title":"Tweet sarcasm detection using deep neural network","authors":"Zhang","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2020] Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. Demo- graphics should not be the reason of toxicity: Mitigating discrimination in text classifications with instance weight- ing. In ACL, 2020.","title":"Demographics should not be the reason of toxicity: Mitigating discrimination in text classifications with instance weighting","authors":"Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, Tiejun Zhao","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2023] Fan Zhou, Yuzhou Mao, Liu Yu, Yi Yang, and Ting Zhong. Causal-Debias: Unifying debiasing in pretrained language models and fine-tuning via causal in- variant learning. In ACL, 2023.","title":"Causal-Debias: Unifying debiasing in pretrained language models and fine-tuning via causal invariant learning","authors":"Fan Zhou, Yuzhou Mao, Liu Yu, Yi Yang, Ting Zhong","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhu et al., 2022] Yongchun Zhu, Qiang Sheng, Juan Cao, Shuokai Li, Danding Wang, and Fuzhen Zhuang. Gener- alizing to the future: Mitigating entity bias in fake news detection. In SIGIR, 2022.","title":"Generalizing to the future: Mitigating entity bias in fake news detection","authors":"Yongchun Zhu, Qiang Sheng, Juan Cao, Shuokai Li, Danding Wang, Fuzhen Zhuang","year":"2022","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2024a] Zhihong Zhu, Xuxin Cheng, Guimin Hu, Yaowei Li, Zhiqi Huang, and Yuexian Zou. Towards multi- modal sarcasm detection via disentangled multi-grained multi-modal distilling. In COLING, 2024.","title":"Towards multimodal sarcasm detection via disentangled multi-grained multi-modal distilling","authors":"Zhu","year":"2024","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu et al., 2024b] Zhihong Zhu, Xuxin Cheng, Hongxiang Li, Yaowei Li, and Yuexian Zou. Dance with labels: Dual- heterogeneous label graph interaction for multi-intent spo- ken language understanding. In WSDM, 2024.","title":"Dance with labels: Dualheterogeneous label graph interaction for multi-intent spoken language understanding","authors":"Zhu","year":"2024","venue":"WSDM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}