{"type":"ScholarlyArticle","title":"Natural Language-centered Inference Network for Multi-modal Fake News Detection","authors":[{"name":"Qiang Zhang","slug":"qiang-zhang","given":"Qiang","family":"Zhang","affiliation":"University of Science and Technology of China , China"},{"name":"Jiawei Liu","slug":"jiawei-liu","given":"Jiawei","family":"Liu","affiliation":"University of Science and Technology of China , China"},{"name":"Fanrui Zhang","slug":"fanrui-zhang","given":"Fanrui","family":"Zhang","affiliation":"University of Science and Technology of China , China"},{"name":"Jingyi Xie","slug":"jingyi-xie","given":"Jingyi","family":"Xie","affiliation":"University of Science and Technology of China , China"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"University of Science and Technology of China , China"}],"abstract":"The proliferation of fake news with image and text in the internet has triggered widespread concern. Existing research has made important contributions in cross-modal information interaction and fusion, but fails to fundamentally address the modality gap among news image, text, and news-related external knowledge representations. In this paper, we propose a novel Natural Language-centered Inference Network (NLIN) for multi-modal fake news detection by aligning multi-modal news content with the natural language space and introducing an encoder-decoder architecture to fully comprehend the news in-context. Specifically, we first unify multi-modal news content into textual modality by converting news images and news-related external knowledge into plain textual content. Then, we design a multi-modal feature reasoning module, which consists of a multi-modal encoder, a unified-modal context encoder and an inference decoder with prompt phrase. This framework not only fully extracts the latent representation of cross-modal news content, but also utilizes the prompt phrase to stimulate the powerful in-context learning ability of the pre-trained large language model to reason about the truthfulness of the news content. In addition, to support the research in the field of multi-modal fake news detection, we produce a challenging large scale, multi-platform, multi-domain multi-modal Chinese Fake News Detection (CFND) dataset. Extensive experiments show that our CFND dataset is challenging and the proposed NLIN outperforms state-of-the-art methods.","doi":"10.24963/ijcai.2024/281","url":"https://ijcai.org/proceedings/2024/281","pdf":"https://ijcai.org/proceedings/2024/0281.pdf","markdown":"https://ijcai.org/proceedings/2024/281.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/281","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Data Mining","track":"Main Track","number":281,"pages":{"first":2542,"last":2550},"keywords":["Multidisciplinary Topics and Applications: MTA: News and media","Data Mining: DM: Mining text, web, social media"],"videos":[],"references":[{"position":1,"raw":"[Abdelnabi et al., 2022] Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. Open-domain, content-based, multi- modal fact-checking of out-of-context images via online resources. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14940- 14949, 2022.","title":"Open-domain, content-based, multimodal fact-checking of out-of-context images via online resources","authors":"Sahar Abdelnabi, Rakibul Hasan, Mario Fritz","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boididou et al., 2018] Christina Boididou, Symeon Pa- padopoulos, Markos Zampoglou, Lazaros Apostolidis, Olga Papadopoulou, and Yiannis Kompatsiaris. Detec- tion and visualization of misleading content on twitter. In- ternational Journal of Multimedia Information Retrieval, 7(1):71-86, 2018.","title":"Detection and visualization of misleading content on twitter","authors":"Christina Boididou, Symeon Papadopoulos, Markos Zampoglou, Lazaros Apostolidis, Olga Papadopoulou, Yiannis Kompatsiaris","year":"2018","venue":"International Journal of Multimedia Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2022a] Yixuan Chen, Dongsheng Li, Peng Zhang, Jie Sui, Qin Lv, Lu Tun, and Li Shang. Cross- modal ambiguity learning for multimodal fake news de- tection. In Proceedings of the ACM Web Conference 2022, pages 2897-2905, 2022.","title":"Crossmodal ambiguity learning for multimodal fake news detection","authors":"Chen","year":"2022","venue":"Proceedings of the ACM Web Conference 2022","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022b] Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Yin Fang, Jeff Z Pan, Ningyu Zhang, and Wen Zhang. Lako: Knowledge-driven visual ques- tion answering via late knowledge-to-text injection. In Proceedings of the 11th International Joint Conference on Knowledge Graphs, pages 20-29, 2022.","title":"Lako: Knowledge-driven visual question answering via late knowledge-to-text injection","authors":"Chen","year":"2022","venue":"Proceedings of the 11th International Joint Conference on Knowledge Graphs","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.","title":"Scaling instruction-finetuned language models","authors":"Chung Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"EasyOCR, ] EasyOCR. ocr. https://github.com/JaidedAI/ EasyOCR.","title":"EasyOCR. ocr","authors":"Easyocr","year":null,"venue":null,"doi":null,"url":"https://github.com/JaidedAI/EasyOCR","internal_url":null},{"position":7,"raw":"[Han et al., 2021] Yi Han, Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie. Knowledge en- hanced multi-modal fake news detection. arXiv preprint arXiv:2108.04418, 2021.","title":"Knowledge enhanced multi-modal fake news detection","authors":"Yi Han, Amila Silva, Ling Luo, Shanika Karunasekera, Christopher Leckie","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2017] Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In Proceedings of the 25th ACM International Conference on Multimedia, pages 795-816, 2017.","title":"Multimodal fusion with recurrent neural networks for rumor detection on microblogs","authors":"Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, Jiebo Luo","year":"2017","venue":"Proceedings of the 25th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Khattar et al., 2019] Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. Mvae: Multimodal variational autoencoder for fake news detection. In The World Wide Web Conference, pages 2915-2921, 2019.","title":"Mvae: Multimodal variational autoencoder for fake news detection","authors":"Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, Vasudeva Varma","year":"2019","venue":"The World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.","title":"Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation","authors":"Li","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2023] Jiawei Liu, Jingyi Xie, Yang Wang, and Zheng-Jun Zha. Adaptive texture and spectrum clue min- ing for generalizable face forgery detection. IEEE Trans- actions on Information Forensics and Security, 2023.","title":"Adaptive texture and spectrum clue mining for generalizable face forgery detection","authors":"Jiawei Liu, Jingyi Xie, Yang Wang, Zheng-Jun Zha","year":"2023","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2015] Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, and Kam-Fai Wong. Detect rumors using time series of social context information on microblogging websites. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1751- 1754, 2015.","title":"Detect rumors using time series of social context information on microblogging websites","authors":"Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, Kam-Fai Wong","year":"2015","venue":"Proceedings of the 24th ACM International on Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nan et al., 2021] Qiong Nan, Juan Cao, Yongchun Zhu, Yanyan Wang, and Jintao Li. Mdfend: Multi-domain fake news detection. In Proceedings of the 30th ACM Interna- tional Conference on Information & Knowledge Manage- ment, pages 3343-3347, 2021.","title":"Mdfend: Multi-domain fake news detection","authors":"Qiong Nan, Juan Cao, Yongchun Zhu, Yanyan Wang, Jintao Li","year":"2021","venue":"Proceedings of the 30th ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Narayan et al., 2022] Kartik Narayan, Harsh Agarwal, Surbhi Mittal, Kartik Thakral, Suman Kundu, Mayank Vatsa, and Richa Singh. Desi: Deepfake source identifier for social media. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 2858-2867, 2022.","title":"Desi: Deepfake source identifier for social media","authors":"Kartik Narayan, Harsh Agarwal, Surbhi Mittal, Kartik Thakral, Suman Kundu, Mayank Vatsa, Richa Singh","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pérez-Rosas et al., 2017] Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. Automatic detection of fake news. arXiv preprint arXiv:1708.07104, 2017.","title":"Automatic detection of fake news","authors":"Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, Rada Mihalcea","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shu et al., 2020] Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. Fakenewsnet: A data repository with news content, social context, and spa- tiotemporal information for studying fake news on social media. Big data, 8(3):171-188, 2020.","title":"Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media","authors":"Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, Huan Liu","year":"2020","venue":"Big data","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Singhal et al., 2019a] Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin'ichi Satoh. Spotfake: A multi-modal framework for fake news detection. In 2019 IEEE fifth International Con- ference on Multimedia Big Data (BigMM), pages 39-47. IEEE, 2019.","title":"Spotfake: A multi-modal framework for fake news detection","authors":"Singhal","year":"2019","venue":"IEEE fifth International Conference on Multimedia Big Data (BigMM)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Singhal et al., 2019b] Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin'ichi Satoh. Spotfake: A multi-modal framework for fake news detection. In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pages 39- 47, 2019.","title":"Spotfake: A multi-modal framework for fake news detection","authors":"Singhal","year":"2019","venue":"IEEE Fifth International Conference on Multimedia Big Data (BigMM)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Singhal et al., 2022] Shivangi Singhal, Tanisha Pandey, Saksham Mrig, Rajiv Ratn Shah, and Ponnurangam Ku- maraguru. Leveraging intra and inter modality relation- ship for multimodal fake news detection. In Companion Proceedings of the Web Conference 2022, pages 726-734, 2022.","title":"Leveraging intra and inter modality relationship for multimodal fake news detection","authors":"Shivangi Singhal, Tanisha Pandey, Saksham Mrig, Rajiv Ratn Shah, Ponnurangam Ku-Maraguru","year":"2022","venue":"Companion Proceedings of the Web Conference 2022","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2021] Mengzhu Sun, Xi Zhang, Jianqiang Ma, and Yazheng Liu. Inconsistency matters: A knowledge- guided dual-inconsistency network for multi-modal rumor detection. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 1412-1423, 2021.","title":"Inconsistency matters: A knowledgeguided dual-inconsistency network for multi-modal rumor detection","authors":"Sun","year":"2021","venue":"Findings of the Association for Computational Linguistics: EMNLP 2021","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tseng et al., 2022] Yu-Wun Tseng, Hui-Kuo Yang, Wei- Yao Wang, and Wen-Chih Peng. Kahan: Knowledge- aware hierarchical attention network for fake news detec- tion on social media. In Companion Proceedings of the Web Conference 2022, pages 868-875, 2022.","title":"Kahan: Knowledgeaware hierarchical attention network for fake news detection on social media","authors":"Yu-Wun Tseng, Hui-Kuo Yang, Wei-Yao Wang, Wen-Chih Peng","year":"2022","venue":"Companion Proceedings of the Web Conference 2022","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vrandečić and Krötzsch, 2014] Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowl- edgebase. Communications of the ACM, 57(10):78-85, 2014.","title":"Wikidata: a free collaborative knowledgebase","authors":"Krötzsch Vrandečić, Denny Vrandečić, Markus Krötzsch","year":"2014","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2021] Yaqing Wang, Fenglong Ma, Haoyu Wang, Kishlay Jha, and Jing Gao. Multimodal emergent fake news detection via meta neural process networks. In Proceedings of the 27th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, pages 3708-3716, 2021.","title":"Multimodal emergent fake news detection via meta neural process networks","authors":"Wang","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2015] Ke Wu, Song Yang, and Kenny Q Zhu. False rumors detection on sina weibo by propagation struc- tures. In 2015 IEEE 31st International Conference on Data Engineering, pages 651-662. IEEE, 2015.","title":"False rumors detection on sina weibo by propagation structures","authors":"Ke Wu, Song Yang, Kenny Q Zhu","year":"2015","venue":"IEEE 31st International Conference on Data Engineering","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2021] Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, and Zhen Xu. Multimodal fusion with co- attention networks for fake news detection. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 2560-2569, 2021.","title":"Multimodal fusion with coattention networks for fake news detection","authors":"Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, Zhen Xu","year":"2021","venue":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ying et al., 2023] Qichao Ying, Xiaoxiao Hu, Yangming Zhou, Zhenxing Qian, Dan Zeng, and Shiming Ge. Boot- strapping multi-view representations for fake news detec- tion, 2023.","title":"Bootstrapping multi-view representations for fake news detection","authors":"Qichao Ying, Xiaoxiao Hu, Yangming Zhou, Zhenxing Qian, Dan Zeng, Shiming Ge","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representa- tions in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.","title":"Vinvl: Revisiting visual representations in vision-language models","authors":"Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2023a] Fanrui Zhang, Jiawei Liu, Qiang Zhang, Esther Sun, Jingyi Xie, and Zheng-Jun Zha. Ecenet: Explainable and context-enhanced network for muti-modal fact verification. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1231-1240, 2023.","title":"Ecenet: Explainable and context-enhanced network for muti-modal fact verification","authors":"Zhang","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2023b] Qiang Zhang, Jiawei Liu, Fanrui Zhang, Jingyi Xie, and Zheng-Jun Zha. Hierarchical se- mantic enhancement network for multimodal fake news detection. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3424-3433, 2023.","title":"Hierarchical semantic enhancement network for multimodal fake news detection","authors":"Zhang","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2020] Xinyi Zhou, Jindi Wu, and Reza Za- farani. : Similarity-aware multi-modal fake news detec- tion. In Advances in Knowledge Discovery and Data Min- ing: 24th Pacific-Asia Conference, PAKDD 2020, Singa- pore, May 11-14, 2020, Proceedings, Part II, pages 354- 367. Springer, 2020.","title":"Similarity-aware multi-modal fake news detection","authors":"Zhou","year":"2020","venue":"Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Singapore","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2022] Yangming Zhou, Qichao Ying, Zhenxing Qian, Sheng Li, and Xinpeng Zhang. Multimodal fake news detection via clip-guided learning. arXiv preprint arXiv:2205.14304, 2022.","title":"Multimodal fake news detection via clip-guided learning","authors":"Yangming Zhou, Qichao Ying, Zhenxing Qian, Sheng Li, Xinpeng Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zubiaga et al., 2017] Arkaitz Zubiaga, Maria Liakata, and Rob Procter. Exploiting context for rumour detection in social media. In Social Informatics: 9th International Conference, SocInfo 2017, Oxford, UK, September 13- 15, 2017, Proceedings, Part I 9, pages 109-123. Springer, 2017.","title":"Exploiting context for rumour detection in social media","authors":"Zubiaga","year":"2017","venue":"Social Informatics: 9th International Conference, SocInfo 2017","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}