{"type":"ScholarlyArticle","title":"Culturally-aware Image Captioning","authors":[{"name":"Youngsik Yun","slug":"youngsik-yun","given":"Youngsik","family":"Yun","affiliation":"Department of Artificial Intelligence , Dongguk University"}],"abstract":"The primary research challenge lies in mitigating and measuring geographical and demographic biases in generative models, which is crucial for ensuring fairness in AI applications. Existing models trained on web-crawled datasets like LAION-400M often perpetuate harmful stereotypes and biases, especially concerning minority groups or less-represented regions. To address this, I proposed a framework called CIC (Culturally-aware Image Caption) to generate culturally-aware image captions. This framework leverages visual question answering (VQA) to extract cultural visual elements from images. It prompts both caption prompts and cultural visual elements to generate culturally-aware captions using large language models (LLMs). Human evaluations confirm the effectiveness of our approach in depicting cultural information accurately.\r\n\r\nTwo key future directions are outlined. First, current image caption evaluation methods are inadequate for assessing culturally-aware captions, necessitating the development of new evaluation metrics leveraging cultural datasets and representations. Second, ethical considerations, particularly concerning stereotypes embedded in existing models, demand consensus and standards development through diverse cultural perspectives. Addressing these challenges is vital for the responsible deployment of AI technologies in diverse real-world contexts.","doi":"10.24963/ijcai.2024/975","url":"https://ijcai.org/proceedings/2024/975","pdf":"https://ijcai.org/proceedings/2024/0975.pdf","markdown":"https://ijcai.org/proceedings/2024/975.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/975","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":975,"pages":{"first":8520,"last":8521},"keywords":["DC: AI Ethics, Trust, Fairness","DC: Computer Vision","DC: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Garcia et al., 2023] Noa Garcia, Yusuke Hirota, Yankun Wu, and Yuta Nakashima. Uncurated image-text datasets: Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6957-6966, 2023.","title":"Uncurated image-text datasets: Shedding light on demographic bias","authors":"Noa Garcia, Yusuke Hirota, Yankun Wu, Yuta Nakashima","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Halpern, 1955] Ben Halpern. The dynamic elements of cul- ture. Ethics, 65(4):235-249, 1955.","title":"The dynamic elements of culture","authors":"Ben Halpern","year":"1955","venue":"Ethics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hessel et al., 2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.","title":"Clipscore: A reference-free evaluation metric for image captioning","authors":"Hessel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hirota et al., 2023] Yusuke Hirota, Yuta Nakashima, and Noa Garcia. Model-agnostic gender debiased image cap- tioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15191- 15200, 2023.","title":"Model-agnostic gender debiased image captioning","authors":"Hirota","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ignat et al., 2024] Oana Ignat, Longju Bai, Joan Nwatu, and Rada Mihalcea. Annotations on a budget: Leveraging geo- data similarity to balance model performance and annota- tion cost. arXiv preprint arXiv:2403.07687, 2024.","title":"Annotations on a budget: Leveraging geodata similarity to balance model performance and annotation cost","authors":"Oana Ignat, Longju Bai, Joan Nwatu, Rada Mihalcea","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Liu et al., 2021] Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. arXiv preprint arXiv:2109.13238, 2021.","title":"Visually grounded reasoning across languages and cultures","authors":"Fangyu Liu, Emanuele Bugliarello, Maria Edoardo, Siva Ponti, Nigel Reddy, Desmond Collier, Elliott","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2024] Zhixuan Liu, Peter Schaldenbrand, Beverley-Claire Okogwu, Wenxuan Peng, Youngsik Yun, Andrew Hundt, Jihie Kim, and Jean Oh. Scoft: Self- contrastive fine-tuning for equitable image generation. arXiv preprint arXiv:2401.08053, 2024.","title":"Scoft: Selfcontrastive fine-tuning for equitable image generation","authors":"Liu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mehrabi et al., 2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.","title":"A survey on bias and fairness in machine learning","authors":"Mehrabi","year":"2021","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Schuhmann et al., 2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clay- ton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.","title":"Laion-400m: Open dataset of clip-filtered 400 million image-text pairs","authors":"Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, Aran Komatsuzaki","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Tao et al., 2023] Yan Tao, Olga Viberg, Ryan S Baker, and Rene F Kizilcec. Auditing and mitigating cultural bias in llms. arXiv preprint arXiv:2311.14096, 2023.","title":"Auditing and mitigating cultural bias in llms","authors":"Yan Tao, Olga Viberg, Ryan S Baker, Rene F Kizilcec","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Wang et al., 2022] Jianfeng Wang, Zhengyuan Yang, Xi- aowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to- text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022.","title":"Git: A generative image-totext transformer for vision and language","authors":"Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Yin et al., 2023] Da Yin, Feng Gao, Govind Thattai, Michael Johnston, and Kai-Wei Chang. Givl: Improving geographical inclusivity of vision-language models with pre-training methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951-10961, 2023.","title":"Givl: Improving geographical inclusivity of vision-language models with pre-training methods","authors":"Feng Da Yin, Govind Gao, Michael Thattai, Kai-Wei Johnston, Chang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Yun and Kim, 2024] Youngsik Yun and Jihie Kim. Cic: A framework for culturally-aware image captioning. arXiv preprint arXiv:2402.05374, 2024.","title":"Cic: A framework for culturally-aware image captioning","authors":"Yun, Kim ;, Youngsik Yun, Jihie Kim","year":"2024","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2024/180"},{"position":16,"raw":"[Zhao et al., 2021] Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 14830- 14840, 2021.","title":"Understanding and evaluating racial biases in image captioning","authors":"Dora Zhao, Angelina Wang, Olga Russakovsky","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}