{"type":"ScholarlyArticle","title":"Beyond What If: Advancing Counterfactual Text Generation with Structural Causal Modeling","authors":[{"name":"Ziao Wang","slug":"ziao-wang","given":"Ziao","family":"Wang","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Shenzhen , China"},{"name":"Xiaofeng Zhang","slug":"xiaofeng-zhang","given":"Xiaofeng","family":"Zhang","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Shenzhen , China"},{"name":"Hongwei Du","slug":"hongwei-du","given":"Hongwei","family":"Du","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Shenzhen , China"}],"abstract":"Exploring the realms of counterfactuals, this paper introduces a versatile approach in text generation using structural causal models (SCM), broadening the scope beyond traditional singular causal studies to encompass complex, multi-layered relationships. To comprehensively explore these intricate, multi-layered causal relationships in text generation, we introduce a generalized approach based on the structural causal model (SCM), adept at handling complex causal interactions in a spectrum ranging from everyday stories to financial reports.Specifically, our method begins by disentangling each component of the text into pairs of latent variables, representing elements that remain unchanged and those subject to variation. Subsequently, counterfactual interventions are applied to these latent variables, facilitating the generation of outcomes that are influenced by complex causal dynamics. Extensive experiments have been conducted on both a public story generation dataset and a specially constructed dataset in the financial domain. The experimental results demonstrate that our approach achieves state-of-the-art performance across a range of automatic and human evaluation criteria, underscoring its effectiveness and versatility in diverse text generation contexts.","doi":"10.24963/ijcai.2024/721","url":"https://ijcai.org/proceedings/2024/721","pdf":"https://ijcai.org/proceedings/2024/0721.pdf","markdown":"https://ijcai.org/proceedings/2024/721.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/721","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":721,"pages":{"first":6522,"last":6530},"keywords":["Machine Learning: ML: Causality","Natural Language Processing: NLP: Language generation","Natural Language Processing: NLP: Applications"],"videos":[],"references":[{"position":1,"raw":"[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proc. of ICML, 2018.","title":"Mutual information neural estimation","authors":"Belghazi","year":"2018","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2022] Jiangjie Chen, Chun Gan, Sijie Cheng, Hao Zhou, Yanghua Xiao, and Lei Li. Unsupervised edit- ing for counterfactual stories. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10473-10481, 2022.","title":"Unsupervised editing for counterfactual stories","authors":"Jiangjie Chen, Chun Gan, Sijie Cheng, Hao Zhou, Yanghua Xiao, Lei Li","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dang-Nhu, 2021] Raphaël Dang-Nhu. Evaluating disen- tanglement of structured representations. arXiv preprint arXiv:2101.04041, 2021.","title":"Evaluating disentanglement of structured representations","authors":"Raphaël Dang-Nhu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feder et al., 2022] Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, et al. Causal in- ference in natural language processing: Estimation, pre- diction, interpretation and beyond. Transactions of the As- sociation for Computational Linguistics, 10:1138-1158, October 2022.","title":"Causal inference in natural language processing: Estimation, prediction, interpretation and beyond","authors":"Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar","year":"2022","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fern and Pope, 2021] Xiaoli Fern and Quintin Pope. Text counterfactuals via latent optimization and shapley-guided search. In Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pages 5578-5593, 2021.","title":"Text counterfactuals via latent optimization and shapley-guided search","authors":"Fern, Pope, Xiaoli Fern, Quintin Pope","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hao et al., 2021] Changying Hao, Liang Pang, Yanyan Lan, Yan Wang, Jiafeng Guo, and Xueqi Cheng. Sketch and customize: A counterfactual story generator. In Proc. of AAAI, 2021.","title":"Sketch and customize: A counterfactual story generator","authors":"Changying Hao, Liang Pang, Yanyan Lan, Yan Wang, Jiafeng Guo, Xueqi Cheng","year":"2021","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu and Li, 2021] Zhiting Hu and Li Erran Li. A causal lens for controllable text generation. Advances in Neural Infor- mation Processing Systems, 34:24941-24955, 2021.","title":"A causal lens for controllable text generation","authors":"Li ; ] Zhiting Hu, Li Hu, Li Erran","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2021] Yi-Chong Huang, Xia-Chong Feng, Xiao-Cheng Feng, and Bing Qin. The factual inconsis- tency problem in abstractive text summarization: A sur- vey. CoRR, 2021.","title":"The factual inconsistency problem in abstractive text summarization: A survey","authors":"Yi-Chong Huang, Xia-Chong Feng, Xiao-Cheng Feng, Bing Qin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khrulkov et al., 2021] Valentin Khrulkov, Leyla Mir- vakhabova, Ivan Oseledets, and Artem Babenko. Disen- tangled representations from non-disentangled models, 2021.","title":"Disentangled representations from non-disentangled models","authors":"Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, Artem Babenko","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kusner et al., 2015] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to docu- ment distances. In Proc. of ICML, 2015.","title":"From word embeddings to document distances","authors":"Matt Kusner, Yu Sun, Nicholas Kolkin, Kilian Weinberger","year":"2015","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2020] Wonkwang Lee, Donggyun Kim, Se- unghoon Hong, and Honglak Lee. High-fidelity synthe- sis with disentangled representation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision -ECCV 2020, pages 157-174, Cham, 2020. Springer International Publishing.","title":"High-fidelity synthesis with disentangled representation","authors":"Lee","year":"2020","venue":"Computer Vision -ECCV 2020","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proc. of ACL, 2020.","title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin, 2004] Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, 2004.","title":"ROUGE: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Qi Liu, Matt Kusner, and Phil Blunsom. Counterfactual data augmentation for neural machine translation. In Proc. of NAACL, 2021.","title":"Counterfactual data augmentation for neural machine translation","authors":"Liu","year":"2021","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2017] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive at- tention via a visual sentinel for image captioning. In Proc. of CVPR, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2016] Zhiyi Luo, Yuchen Sha, Kenny Q. Zhu, Seung-won Hwang, and Zhongyuan Wang. Commonsense causal reasoning between short texts. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, page 421-430, 2016.","title":"Commonsense causal reasoning between short texts","authors":"Zhiyi Luo, Yuchen Sha, Kenny Q Zhu, Seung-Won Hwang, Zhongyuan Wang","year":"2016","venue":"Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pearl et al., 2016] Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.","title":"Causal inference in statistics: A primer","authors":"Pearl","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qin et al., 2019] Lianhui Qin, Antoine Bosselut, Ari Holtz- man, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. Counterfactual story reasoning and generation. In Proc. of EMNLP, 2019.","title":"Counterfactual story reasoning and generation","authors":"Qin","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qin et al., 2020] Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. Back to the future: Unsupervised backprop-based decoding for counterfac- tual and abductive commonsense reasoning. In Proc. of EMNLP, 2020.","title":"Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning","authors":"Qin","year":"2020","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ren et al., 2022] Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Learning disentangled representation by exploiting pretrained generative models: A contrastive learning view. In International Conference on Learning Representations, 2022.","title":"Learning disentangled representation by exploiting pretrained generative models: A contrastive learning view","authors":"Ren","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Roese, 1997] Neal J Roese. Counterfactual thinking. Psy- chological bulletin, 1997.","title":"Counterfactual thinking","authors":"Roese Neal","year":"1997","venue":"Psychological bulletin","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schölkopf et al., 2021] Bernhard Schölkopf, Francesco Lo- catello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbren- ner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 2021.","title":"Toward causal representation learning","authors":"Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, Yoshua Bengio","year":"2021","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vasilakes et al., 2022] Jake Vasilakes, Chrysoula Zerva, Makoto Miwa, and Sophia Ananiadou. Learning disentan- gled representations of negation and uncertainty. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8380-8397, Dublin, Ireland, May 2022. Association for Computational Linguistics.","title":"Learning disentangled representations of negation and uncertainty","authors":"Vasilakes","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Voita et al., 2018] Elena Voita, Pavel Serdyukov, Rico Sen- nrich, and Ivan Titov. Context-aware neural machine trans- lation learns anaphora resolution. In Proc. of ACL, 2018.","title":"Context-aware neural machine translation learns anaphora resolution","authors":"Elena Voita, Pavel Serdyukov, Rico Sennrich, Ivan Titov","year":"2018","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2019] Ke Wang, Hang Hua, and Xiaojun Wan. Controllable unsupervised text attribute transfer via edit- ing entangled latent representation. Advances in Neural Information Processing Systems, 32, 2019.","title":"Controllable unsupervised text attribute transfer via editing entangled latent representation","authors":"Ke Wang, Hang Hua, Xiaojun Wan","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2021] Tongshuang Wu, Marco Tulio Ribeiro, Jef- frey Heer, and Daniel Weld. Polyjuice: Generating coun- terfactuals for explaining, evaluating, and improving mod- els. In Proc. of ACL, 2021.","title":"Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models","authors":"Wu","year":"2021","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yamada et al., 2019] Masanori Yamada, Heecheol Kim, Kosuke Miyoshi, and Hiroshi Yamakawa. Favae: Se- quence disentanglement using information bottleneck principle, 2019.","title":"Favae: Sequence disentanglement using information bottleneck principle","authors":"Masanori Yamada, Heecheol Kim, Kosuke Miyoshi, Hiroshi Yamakawa","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2021] Yazheng Yang, Boyuan Pan, Deng Cai, and Huan Sun. Topnet: Learning from neural topic model to generate long stories. In Proc. of KDD, 2021.","title":"Topnet: Learning from neural topic model to generate long stories","authors":"Yang","year":"2021","venue":"Proc. of KDD","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zellers et al., 2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? In Proc. of ACL, 2019.","title":"HellaSwag: Can a machine really finish your sentence?","authors":"Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi","year":"2019","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang* et al., 2020] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In Proc. of ICLR, 2020.","title":"Bertscore: Evaluating text generation with bert","authors":"Tianyi Zhang, *, Varsha Kishore, *, Felix Wu, *, Kilian Q Weinberger, Yoav Artzi","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2023] Jiageng Zhu, Hanchen Xie, and Wael Abd- Almageed. Sw-vae: Weakly supervised learn disentangled representation vianbsp;latent factor swapping. In Com- puter Vision -ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, page 73-87, Berlin, Heidelberg, 2023. Springer-Verlag.","title":"Sw-vae: Weakly supervised learn disentangled representation vianbsp;latent factor swapping","authors":"Zhu","year":"2022","venue":"Computer Vision -ECCV 2022 Workshops","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}