{"type":"ScholarlyArticle","title":"CIC: A Framework for Culturally-Aware Image Captioning","authors":[{"name":"Youngsik Yun","slug":"youngsik-yun","given":"Youngsik","family":"Yun","affiliation":"Department of Computer Science and Artificial Intelligence , Dongguk University"},{"name":"Jihie Kim","slug":"jihie-kim","given":"Jihie","family":"Kim","affiliation":"Division of AI Software Convergence , Dongguk University"}],"abstract":"Image Captioning generates descriptive sentences from images using Vision-Language Pre-trained models (VLPs) such as BLIP, which has improved greatly. However, current methods lack the generation of detailed descriptive captions for the cultural elements depicted in the images, such as the traditional clothing worn by people from Asian cultural groups. In this paper, we propose a new framework, Culturally-aware Image Captioning (CIC), that generates captions and describes cultural elements extracted from cultural visual elements in images representing cultures. Inspired by methods combining visual modality and Large Language Models (LLMs) through appropriate prompts, our framework (1) generates questions based on cultural categories from images,  (2) extracts cultural visual elements from Visual Question Answering (VQA) using generated questions, and (3) generates culturally-aware captions using LLMs with the prompts. Our human evaluation conducted on 45 participants from 4 different cultural groups with a high understanding of the corresponding culture shows that our proposed framework generates more culturally descriptive captions when compared to the image captioning baseline based on VLPs. Resources can be found at https://shane3606.github.io/cic.","doi":"10.24963/ijcai.2024/180","url":"https://ijcai.org/proceedings/2024/180","pdf":"https://ijcai.org/proceedings/2024/0180.pdf","markdown":"https://ijcai.org/proceedings/2024/180.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/180","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":180,"pages":{"first":1625,"last":1633},"keywords":["Computer Vision: CV: Scene analysis and understanding","Computer Vision: CV: Bias, fairness and privacy","Computer Vision: CV: Vision, language and reasoning"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382-398. Springer, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"Computer Vision-ECCV 2016: 14th European Conference","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerjee et al., 2020] Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. Weaqa: Weak supervision via captions for visual question answering. arXiv preprint arXiv:2012.02356, 2020.","title":"Weaqa: Weak supervision via captions for visual question answering","authors":"Banerjee","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Birhane et al., 2021] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misog- yny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021.","title":"Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misog-yny, pornography, and malignant stereotypes","authors":"Birhane","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Changpinyo et al., 2022] Soravit Changpinyo, Doron Kuk- liansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Sori- cut. All you may need for vqa are image captions. arXiv preprint arXiv:2205.01883, 2022.","title":"Soravit Changpinyo","authors":"Changpinyo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2022] Jiaxian Guo, Junnan Li, Dongxu Li, An- thony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero- shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846, 2022.","title":"From images to textual prompts: Zeroshot vqa with frozen large language models","authors":"Guo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Halpern, 1955] Ben Halpern. The dynamic elements of cul- ture. Ethics, 65(4):235-249, 1955.","title":"The dynamic elements of culture","authors":"Ben Halpern","year":"1955","venue":"Ethics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hershcovich et al., 2022] Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, et al. Chal- lenges and strategies in cross-cultural nlp. arXiv preprint arXiv:2203.10020, 2022.","title":"Challenges and strategies in cross-cultural nlp","authors":"Daniel Hershcovich, Stella Frank, Heather Lent, Mostafa Miryam De Lhoneux, Stephanie Abdou, Emanuele Brandl, Laura Cabello Bugliarello, Ilias Piqueras, Ruixiang Chalkidis, Cui","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hessel et al., 2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.","title":"Clipscore: A reference-free evaluation metric for image captioning","authors":"Hessel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ipeirotis, 2010] Panagiotis G Ipeirotis. Demographics of mechanical turk. 2010.","title":"Demographics of mechanical turk","authors":"G Panagiotis, Ipeirotis","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Zhang, 2023] Zhi Li and Yin Zhang. Cultural con- cept adaptation on multimodal reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 262-276, 2023.","title":"Cultural concept adaptation on multimodal reasoning","authors":"Li, Zhang ; ] Zhi Li, Yin Zhang","year":"2023","venue":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. arXiv preprint arXiv:2109.13238, 2021.","title":"Visually grounded reasoning across languages and cultures","authors":"Fangyu Liu, Emanuele Bugliarello, Maria Edoardo, Siva Ponti, Nigel Reddy, Desmond Collier, Elliott","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2022] Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqin Yang. Vision-and-language pretrained models: A survey. arXiv preprint arXiv:2204.07356, 2022.","title":"Vision-and-language pretrained models: A survey","authors":"Long","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/773"},{"position":15,"raw":"[Mandal et al., 2021] Abhishek Mandal, Susan Leavy, and Suzanne Little. Dataset diversity: measuring and mitigat- ing geographical bias in image search and retrieval. In Proceedings of the 1st International Workshop on Trust- worthy AI for Multimedia Computing, pages 19-25, 2021.","title":"Dataset diversity: measuring and mitigating geographical bias in image search and retrieval","authors":"Abhishek Mandal, Susan Leavy, Suzanne Little","year":"2021","venue":"Proceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mehrabi et al., 2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.","title":"A survey on bias and fairness in machine learning","authors":"Mehrabi","year":"2021","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen et al., 2023] Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907-1917, 2023.","title":"Extracting cultural commonsense knowledge at scale","authors":"Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, Gerhard Weikum","year":"2023","venue":"Proceedings of the ACM Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Plummer et al., 2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collect- ing region-to-phrase correspondences for richer image-to- sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.","title":"Flickr30k entities: Collecting region-to-phrase correspondences for richer image-tosentence models","authors":"Liwei Bryan A Plummer, Chris M Wang, Juan C Cervantes, Julia Caicedo, Svetlana Hockenmaier, Lazebnik","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radford and others, 2021] Radford et al. Learning trans- ferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Radford","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rojas et al., 2022] William A Gaviria Rojas, Sudnya Di- amos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Im- ages representing the geographic and socioeconomic di- versity of the world. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.","title":"The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world","authors":"William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, Cody Coleman","year":"2022","venue":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharma et al., 2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.","title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning","authors":"Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut","year":"2018","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Srinivasan et al., 2021] Krishna Srinivasan, Karthik Ra- man, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443-2449, 2021.","title":"Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning","authors":"Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, Marc Najork","year":"2021","venue":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2020] Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. Compare and reweight: Dis- tinctive image captioning using similar images sets. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 370-386. Springer, 2020.","title":"Compare and reweight: Distinctive image captioning using similar images sets","authors":"Wang","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022a] Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shi- rai, Arvind Narayanan, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. International Journal of Computer Vision, 130(7):1790- 1810, 2022.","title":"Revise: A tool for measuring and mitigating bias in visual datasets","authors":"Wang","year":"2022","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022b] Jianfeng Wang, Zhengyuan Yang, Xi- aowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to- text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022.","title":"Git: A generative image-totext transformer for vision and language","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2022] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Li- juan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 3081-3089, 2022.","title":"An empirical study of gpt-3 for few-shot knowledge-based vqa","authors":"Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, Lijuan Wang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yin et al., 2021] Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. Broaden the vi- sion: Geo-diverse visual commonsense reasoning. arXiv preprint arXiv:2109.06860, 2021.","title":"Broaden the vision: Geo-diverse visual commonsense reasoning","authors":"Liunian Da Yin, Ziniu Harold Li, Nanyun Hu, Kai-Wei Peng, Chang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yin et al., 2022] Da Yin, Hritik Bansal, Masoud Mona- jatipoor, Liunian Harold Li, and Kai-Wei Chang. Ge- omlama: Geo-diverse commonsense probing on mul- tilingual pre-trained language models. arXiv preprint arXiv:2205.12247, 2022.","title":"Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language models","authors":"Hritik Da Yin, Masoud Bansal, Liunian Monajatipoor, Kai-Wei Harold Li, Chang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yin et al., 2023] Da Yin, Feng Gao, Govind Thattai, Michael Johnston, and Kai-Wei Chang. Givl: Improving geographical inclusivity of vision-language models with pre-training methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951-10961, 2023.","title":"Givl: Improving geographical inclusivity of vision-language models with pre-training methods","authors":"Feng Da Yin, Govind Gao, Michael Thattai, Kai-Wei Johnston, Chang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2022] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.","title":"Coca: Contrastive captioners are image-text foundation models","authors":"Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yuan, 2021] Desen Yuan. Language bias in visual ques- tion answering: A survey and taxonomy. arXiv preprint arXiv:2111.08531, 2021.","title":"Desen Yuan. Language bias in visual question answering: A survey and taxonomy","authors":"Yuan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, Kilichbek Hay- darov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594, 2023.","title":"Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}