{"type":"ScholarlyArticle","title":"KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph","authors":[{"name":"Yanbei Jiang","slug":"yanbei-jiang","given":"Yanbei","family":"Jiang","affiliation":"School of Computing and Information Systems , The University of Melbourne"},{"name":"Krista A. Ehinger","slug":"krista-a-ehinger","given":"Krista A.","family":"Ehinger","affiliation":"School of Computing and Information Systems , The University of Melbourne"},{"name":"Jey Han Lau","slug":"jey-han-lau","given":"Jey Han","family":"Lau","affiliation":"School of Computing and Information Systems , The University of Melbourne"}],"abstract":"Exploring the narratives conveyed by fine-art paintings is a challenge in image captioning, where the goal is to generate descriptions that not only precisely represent the visual content but also offer a in-depth interpretation of the artwork's meaning. The task is particularly complex for artwork images due to their diverse interpretations and varied aesthetic principles across different artistic schools and styles. In response to this, we present KALE (Knowledge-Augmented vision-Language model for artwork Elaborations), a novel approach that enhances existing vision-language models by integrating artwork metadata as additional knowledge. KALE incorporates the metadata in two ways: firstly as direct textual input, and secondly through a multimodal heterogeneous knowledge graph. To optimize the learning of graph representations, we introduce a new cross-modal alignment loss that maximizes the similarity between the image and its corresponding metadata. Experimental results demonstrate that KALE achieves strong performance (when evaluated with CIDEr, in particular) over existing state-of-the-art work across several artwork datasets. Source code of the project is available at https://github.com/Yanbei-Jiang/Artwork-Interpretation.","doi":"10.24963/ijcai.2024/848","url":"https://ijcai.org/proceedings/2024/848","pdf":"https://ijcai.org/proceedings/2024/0848.pdf","markdown":"https://ijcai.org/proceedings/2024/848.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/848","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":848,"pages":{"first":7663,"last":7671},"keywords":["Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Application domains: Text, literature and creative language","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Images, movies and visual arts"],"videos":[],"references":[{"position":1,"raw":"[Achlioptas et al., 2021] Panos Achlioptas, Maks Ovs- janikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J Guibas. Artemis: Affective language for visual art. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11569-11579, 2021.","title":"Artemis: Affective language for visual art","authors":"Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, Leonidas J Guibas","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382-398. Springer, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould","year":"2016","venue":"Computer Vision-ECCV 2016: 14th European Conference","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. stat, 1050:21, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2021] Zechen Bai, Yuta Nakashima, and Noa Garcia. Explain me the painting: Multi-topic knowledge- able art description generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5422-5432, 2021.","title":"Explain me the painting: Multi-topic knowledgeable art description generation","authors":"Bai","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceed- ings of the acl workshop on intrinsic and extrinsic evalua- tion measures for machine translation and/or summariza- tion, pages 65-72, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cetinic, 2021] Eva Cetinic. Iconographic image captioning for artworks. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10- 15, 2021, Proceedings, Part III, pages 502-516. Springer, 2021.","title":"Eva Cetinic. Iconographic image captioning for artworks","authors":"Cetinic","year":"2021","venue":"Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cornia et al., 2020] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10578-10587, 2020.","title":"Meshed-memory transformer for image captioning","authors":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In International Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garcia and Vogiatzis, 2018] Noa Garcia and George Vo- giatzis. How to read paintings: semantic art understanding with multi-modal retrieval. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.","title":"Noa Garcia and George Vogiatzis. How to read paintings: semantic art understanding with multi-modal retrieval","authors":"Vogiatzis Garcia","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV) Workshops","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garcia et al., 2020] Noa Garcia, Benjamin Renoust, and Yuta Nakashima. Contextnet: representation and ex- ploration for painting classification and retrieval in con- text. International Journal of Multimedia Information Re- trieval, 9:17-30, 2020.","title":"Contextnet: representation and exploration for painting classification and retrieval in context","authors":"Garcia","year":"2020","venue":"International Journal of Multimedia Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gonthier et al., 2018] Nicolas Gonthier, Yann Gousseau, Said Ladjal, and Olivier Bonfait. Weakly supervised ob- ject detection in artworks. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.","title":"Weakly supervised object detection in artworks","authors":"Nicolas Gonthier, Yann Gousseau, Said Ladjal, Olivier Bonfait","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV) Workshops","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Joulin et al., 2016] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for effi- cient text classification. arXiv preprint arXiv:1607.01759, 2016.","title":"Bag of tricks for efficient text classification","authors":"Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In- ternational journal of computer vision, 123:32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.","title":"Oscar: Object-semantics aligned pre-training for vision-language tasks","authors":"Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2022] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip- connections. arXiv preprint arXiv:2205.12005, 2022.","title":"mplug: Effective and efficient vision-language learning by cross-modal skipconnections","authors":"Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Loshchilov and Hutter, 2018] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Inter- national Conference on Learning Representations, 2018.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lu et al., 2022] Yue Lu, Chao Guo, Xingyuan Dai, and Fei- Yue Wang. Artcap: A dataset for image captioning of fine art paintings. IEEE Transactions on Computational Social Systems, 2022.","title":"Artcap: A dataset for image captioning of fine art paintings","authors":"Yue Lu, Chao Guo, Xingyuan Dai, Fei-Yue Wang","year":"2022","venue":"IEEE Transactions on Computational Social Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ma et al., 2017] Daiqian Ma, Feng Gao, Yan Bai, Yihang Lou, Shiqi Wang, Tiejun Huang, and Ling-Yu Duan. From part to whole: who is behind the painting? In Proceedings of the 25th ACM international conference on Multimedia, pages 1174-1182, 2017.","title":"From part to whole: who is behind the painting?","authors":"Daiqian Ma, Feng Gao, Yan Bai, Yihang Lou, Shiqi Wang, Tiejun Huang, Ling-Yu Duan","year":"2017","venue":"Proceedings of the 25th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sharma et al., 2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.","title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning","authors":"Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sheng and Moens, 2019] Shurong Sheng and Marie- Francine Moens. Generating captions for images of ancient artworks. In Proceedings of the 27th ACM Inter- national Conference on Multimedia, pages 2478-2486, 2019.","title":"Shurong Sheng and Marie-Francine Moens. Generating captions for images of ancient artworks","authors":"Moens Sheng","year":"2019","venue":"Proceedings of the 27th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stefanini et al., 2019] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and Rita Cuc- chiara. Artpedia: A new visual-semantic dataset with vi- sual and contextual sentences in the artistic domain. In Image Analysis and Processing-ICIAP 2019: 20th Inter- national Conference, Trento, Italy, September 9-13, 2019, Proceedings, Part II 20, pages 729-740. Springer, 2019.","title":"Artpedia: A new visual-semantic dataset with visual and contextual sentences in the artistic domain","authors":"Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, Rita Cucchiara","year":"2019","venue":"Image Analysis and Processing-ICIAP 2019: 20th International Conference","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2019] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Hetero- geneous graph attention network. In The world wide web conference, pages 2022-2032, 2019.","title":"Heterogeneous graph attention network","authors":"Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, Philip S Yu","year":"2019","venue":"The world wide web conference","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2021] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervi- sion. arXiv preprint arXiv:2108.10904, 2021.","title":"Simvlm: Simple visual language model pretraining with weak supervision","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2022] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jin- gren Zhou, and Hongxia Yang. Ofa: Unifying architec- tures, tasks, and modalities through a simple sequence-to- sequence learning framework. In International Conference on Machine Learning, pages 23318-23340. PMLR, 2022.","title":"Ofa: Unifying architectures, tasks, and modalities through a simple sequence-tosequence learning framework","authors":"Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu, 2022] Zixun Wu. Artwork interpretation. Master's the- sis, University of Melbourne, 2022.","title":"Artwork interpretation. Master's thesis","authors":"Wu","year":"2022","venue":"Zixun Wu","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wynen et al., 2018] Daan Wynen, Cordelia Schmid, and Julien Mairal. Unsupervised learning of artistic styles with archetypal style analysis. Advances in Neural Information Processing Systems, 31, 2018.","title":"Unsupervised learning of artistic styles with archetypal style analysis","authors":"Daan Wynen, Cordelia Schmid, Julien Mairal","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2019] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10685- 10694, 2019.","title":"Auto-encoding scene graphs for image captioning","authors":"Xu Yang, Kaihua Tang, Hanwang Zhang, Jianfei Cai","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhao and Wu, 2023] Wentian Zhao and Xinxiao Wu. Boost- ing entity-aware image captioning with multi-modal knowledge graph. IEEE Transactions on Multimedia, 2023.","title":"Boosting entity-aware image captioning with multi-modal knowledge graph","authors":"Wu Zhao, Wentian Zhao, Xinxiao Wu","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}