{"type":"ScholarlyArticle","title":"Prompt-enhanced Network for Hateful Meme Classification","authors":[{"name":"Junxi Liu","slug":"junxi-liu","given":"Junxi","family":"Liu","affiliation":"School of Electronics and Information Engineering , South China Normal University , Foshan 528225 , China,"},{"name":"Yanyan Feng","slug":"yanyan-feng","given":"Yanyan","family":"Feng","affiliation":"School of Electronics and Information Engineering , South China Normal University , Foshan 528225 , China,"},{"name":"Jiehai Chen","slug":"jiehai-chen","given":"Jiehai","family":"Chen","affiliation":"School of Electronics and Information Engineering , South China Normal University , Foshan 528225 , China,"},{"name":"Yun Xue","slug":"yun-xue","given":"Yun","family":"Xue","affiliation":"School of Electronics and Information Engineering , South China Normal University , Foshan 528225 , China,"},{"name":"Fenghuan Li","slug":"fenghuan-li","given":"Fenghuan","family":"Li","affiliation":"School of Computer Science and Technology , Guangdong University of Technology , Guangzhou 510006 , China"}],"abstract":"The dynamic expansion of social media has led to an inundation of hateful memes on media platforms, accentuating the growing need for efficient identification and removal. Acknowledging the constraints of conventional multimodal hateful meme classification, which heavily depends on external knowledge and poses the risk of including irrelevant or redundant content, we developed Pen—a prompt-enhanced network framework based on the prompt learning approach. Specifically, after constructing the sequence through the prompt method and encoding it with a language model, we performed region information global  extraction on the encoded sequence for multi-view perception. By capturing global information about inference instances and demonstrations, Pen facilitates category selection by fully leveraging sequence information. This approach significantly improves model classification accuracy. Additionally, to bolster the model's reasoning capabilities in the feature space, we introduced prompt-aware contrastive learning into the framework to improve the quality of sample feature distributions. Through extensive ablation experiments on two public datasets, we evaluate the effectiveness of the Pen framework, concurrently comparing it with state-of-the-art model baselines. Our research findings highlight that Pen surpasses manual prompt methods, showcasing superior generalization and classification accuracy in hateful meme classification tasks. Our code is available at https://github.com/juszzi/Pen.","doi":"10.24963/ijcai.2024/707","url":"https://ijcai.org/proceedings/2024/707","pdf":"https://ijcai.org/proceedings/2024/0707.pdf","markdown":"https://ijcai.org/proceedings/2024/707.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/707","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":707,"pages":{"first":6397,"last":6405},"keywords":["Natural Language Processing: NLP: Language models","Machine Learning: ML: Feature extraction, selection and dimensionality reduction","Natural Language Processing: NLP: Text classification","Machine Learning: ML: Clustering"],"videos":[],"references":[{"position":1,"raw":"[Blaier et al., 2021] Efrat Blaier, Itzik Malkiel, and Lior Wolf. Caption enriched samples for improving hateful memes detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Re- public, 7-11 November, 2021, pages 9350-9358. Associa- tion for Computational Linguistics, 2021.","title":"Caption enriched samples for improving hateful memes detection","authors":"Blaier","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2022] Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. Prompting for multimodal hateful meme classification. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Process- ing, EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022, pages 321-332. Association for Com- putational Linguistics, 2022.","title":"Prompting for multimodal hateful meme classification","authors":"Rui Cao, Roy, Ka-Wei Lee, Wen-Haw Chong, Jing Jiang","year":"2022","venue":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2023] Rui Cao, Ming Shan Hee, Adriel Kuek, Wen-Haw Chong, Roy Ka-Wei Lee, and Jing Jiang. Pro- cap: Leveraging a frozen vision-language model for hate- ful meme detection. In Proceedings of the 31st ACM In- ternational Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023-3 November 2023, pages 5244-5252. ACM, 2023.","title":"Procap: Leveraging a frozen vision-language model for hateful meme detection","authors":"Rui Cao, Ming Shan Hee, Adriel Kuek, Wen-Haw Chong, Roy, Ka-Wei Lee, Jing Jiang","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia, MM 2023","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, pages 4171-4186. Association for Computational Linguistics, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2022] Huaicheng Fang, Fuqing Zhu, Jizhong Han, and Songlin Hu. Multimodal hateful memes detection via image caption supervision. In Pro- ceedings of 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles, Smart- World/UIC/ScalCom/DigitalTwin/PriComp/Meta 2022, Haikou, China, December 15-18, 2022, pages 1530-1537. IEEE, 2022.","title":"Multimodal hateful memes detection via image caption supervision","authors":"Fang","year":"2022","venue":"Proceedings of 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2023] Xinlei He, Savvas Zannettou, Yun Shen, and Yang Zhang. You only prompt once: On the capabili- ties of prompt learning on large language models to tackle toxic content. CoRR, abs/2308.05596, 2023.","title":"You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content","authors":"He","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jian et al., 2022] Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Contrastive learning for prompt-based few-shot language learners. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 5577-5587. Association for Computa- tional Linguistics, 2022.","title":"Contrastive learning for prompt-based few-shot language learners","authors":"Yiren Jian, Chongyang Gao, Soroush Vosoughi","year":"2022","venue":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kärkkäinen and Joo, 2021] Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pages 1547-1557. IEEE, 2021.","title":"Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation","authors":"Joo Kärkkäinen, Kimmo Kärkkäinen, Jungseock Joo","year":"2021","venue":"Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kiela et al., 2019] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bi- transformers for classifying images and text. In Proceed- ings of the 2019 Conference on Visually Grounded Inter- action and Language (ViGIL), NeurIPS 2019 Workshop, Vancouver, Canada, December 13, 2019, 2019.","title":"Supervised multimodal bitransformers for classifying images and text","authors":"Kiela","year":"2019","venue":"Proceedings of the 2019 Conference on Visually Grounded Interaction and Language (ViGIL), NeurIPS 2019 Workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kiela et al., 2020] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In Proceedings of the 2020 Conference on Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.","title":"The hateful memes challenge: Detecting hate speech in multimodal memes","authors":"Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, Davide Testuggine","year":"2020","venue":"Proceedings of the 2020 Conference on Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2021] Roy Ka-Wei Lee, Rui Cao, Ziqing Fan, Jing Jiang, and Wen-Haw Chong. Disentangling hate in online memes. In Proceedings of the MM '21: ACM Mul- timedia Conference, Virtual Event, China, October 20-24, 2021, pages 5138-5147. ACM, 2021.","title":"Disentangling hate in online memes","authors":"Ka-Wei Roy, Rui Lee, Ziqing Cao, Jing Fan, Wen-Haw Jiang, Chong","year":"2021","venue":"Proceedings of the MM '21: ACM Multimedia Conference, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. CoRR, abs/1908.03557, 2019.","title":"Visualbert: A simple and performant baseline for vision and language","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large lan- guage models. In Proceedings of the International Con- ference on Machine Learning, ICML 2023, Honolulu, Hawaii, USA, 23-29 July 2023, volume 202, pages 19730- 19742. PMLR, 2023.","title":"BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models","authors":"Li","year":"2023","venue":"Proceedings of the International Conference on Machine Learning, ICML 2023","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang et al., 2022] Bin Liang, Qinglin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, and Ruifeng Xu. Jointcl: A joint contrastive learning framework for zero-shot stance detec- tion. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 81-91. Association for Computational Linguistics, 2022.","title":"Jointcl: A joint contrastive learning framework for zero-shot stance detection","authors":"Bin Liang, Qinglin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, Ruifeng Xu","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019.","title":"Roberta: A robustly optimized BERT pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2023] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9):195:1-195:35, 2023.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig","year":"2023","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. In Proceedings of the 2019 Conference on Advances in Neu- ral Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurIPS 2019, Vancouver, BC, CanadaDecember, 8-14, 2019, pages 13-23, 2019.","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":"Proceedings of the 2019 Conference on Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mokady et al., 2021] Ron Mokady, Amir Hertz, and Amit H. Bermano. Clipcap: CLIP prefix for image cap- tioning. CoRR, abs/2111.09734, 2021.","title":"Clipcap: CLIP prefix for image captioning","authors":"Ron Mokady, Amir Hertz, Amit H Bermano","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Muennighoff, 2020] Niklas Muennighoff. Vilio: State-of- the-art visio-linguistic models applied to hateful memes. CoRR, abs/2012.07788, 2020.","title":"Niklas Muennighoff. Vilio: State-ofthe-art visio-linguistic models applied to hateful memes","authors":"Muennighoff","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Piccoli et al., 2024] Valentina Piccoli, Andrea Carnaghi, Michele Grassi, Marta Stragà, and Mauro Bianchi. Cor- rigendum to cyberbullying through the lens of social in- fluence: Predicting cyberbullying perpetration from per- ceived peer-norm, cyberspace regulations and ingroup pro- cesses [comput. hum. behav. 102c (2020) 260-273. Com- put. Hum. Behav., 151:108040, 2024.","title":"Corrigendum to cyberbullying through the lens of social influence: Predicting cyberbullying perpetration from perceived peer-norm, cyberspace regulations and ingroup processes [comput","authors":"Valentina Piccoli, Andrea Carnaghi, Michele Grassi, Marta Stragà, Mauro Bianchi","year":"2020","venue":"Comput. Hum. Behav","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pramanick et al., 2021a] Shraman Pramanick, Dimi- tar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. Detecting harmful memes and their targets. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021, pages 2783-2796. Association for Computational Linguistics, 2021.","title":"Detecting harmful memes and their targets","authors":"Pramanick","year":"2021","venue":"Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pramanick et al., 2021b] Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. MOMENTA: A multi- modal framework for detecting harmful memes and their targets. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 4439-4455. Association for Computational Linguistics, 2021.","title":"MOMENTA: A multimodal framework for detecting harmful memes and their targets","authors":"Pramanick","year":"2021","venue":"Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Qu et al., 2023] Yiting Qu, Xinlei He, Shannon Pierson, Michael Backes, Yang Zhang, and Savvas Zannettou. On the evolution of (hateful) memes by means of multimodal contrastive learning. In Proceedings of the 44th IEEE Sym- posium on Security and Privacy, SP 2023, San Francisco, CA, USA, May 21-25, 2023, pages 293-310. IEEE, 2023.","title":"On the evolution of (hateful) memes by means of multimodal contrastive learning","authors":"Yiting Qu, Xinlei He, Shannon Pierson, Michael Backes, Yang Zhang, Savvas Zannettou","year":"2023","venue":"Proceedings of the 44th IEEE Symposium on Security and Privacy, SP 2023","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ren et al., 2017] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell., 39(6):1137-1149, 2017.","title":"Faster R-CNN: towards real-time ob-ject detection with region proposal networks","authors":"Ren","year":"2017","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Suryawanshi et al., 2020] Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mihael Arcan, and Paul Buitelaar. Multimodal meme dataset (multioff) for identi- fying offensive content in image and text. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, TRAC@LREC 2020, Marseille, France, May 2020, pages 32-41. European Language Resources Association (ELRA), 2020.","title":"Multimodal meme dataset (multioff) for identifying offensive content in image and text","authors":"Shardul Suryawanshi, Raja Bharathi, Mihael Chakravarthi, Paul Arcan, Buitelaar","year":"2020","venue":"Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, TRAC@LREC 2020","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[van der Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579-2605, 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Velioglu and Rose, 2020] Riza Velioglu and Jewgeni Rose. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge. CoRR, abs/2012.12975, 2020.","title":"Riza Velioglu and Jewgeni Rose. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge","authors":"Rose Velioglu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2022] Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Y. S. Lam. New intent discovery with pre-training and contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 256- 269. Association for Computational Linguistics, 2022.","title":"New intent discovery with pre-training and contrastive learning","authors":"Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, Albert Y S Lam","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2021] Yi Zhou, Zhenhao Chen, and Huiyuan Yang. Multimodal learning for hateful memes detection. In Proceedings of the 2021 IEEE International Confer- ence on Multimedia & Expo Workshops, ICME Work- shops, Shenzhen, China, July 5-9, 2021, pages 1-6. IEEE, 2021.","title":"Multimodal learning for hateful memes detection","authors":"Yi Zhou, Zhenhao Chen, Huiyuan Yang","year":"2021","venue":"Proceedings of the 2021 IEEE International Conference on Multimedia & Expo Workshops","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu, 2020] Ron Zhu. Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme chal- lenge winning solution. CoRR, abs/2012.08290, 2020.","title":"Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution","authors":"Ron Zhu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}