{"type":"ScholarlyArticle","title":"Knowledge Aware Semantic Concept Expansion for Image-Text Matching","authors":[{"name":"Botian Shi","slug":"botian-shi","given":"Botian","family":"Shi","affiliation":"Beijing Institute of Technology"},{"name":"Lei Ji","slug":"lei-ji","given":"Lei","family":"Ji","affiliation":"Institute of Computing Technology , Chinese Academy of Science , Beijing , China; Natural Language Computing , Microsoft Research Asia , Beijing , China"},{"name":"Pan Lu","slug":"pan-lu","given":"Pan","family":"Lu","affiliation":"University of California , Los Angeles"},{"name":"Zhendong Niu","slug":"zhendong-niu","given":"Zhendong","family":"Niu","affiliation":"Beijing Institute of Technology"},{"name":"Nan Duan","slug":"nan-duan","given":"Nan","family":"Duan","affiliation":"Natural Language Computing , Microsoft Research Asia , Beijing , China"}],"abstract":"Image-text matching is a vital cross-modality task in artificial intelligence and has attracted increasing attention in recent years. Existing works have shown that learning semantic concepts is useful to enhance image representation and can significantly improve the performance of both image-to-text and text-to-image retrieval. However, existing models simply detect semantic concepts from a given image, which are less likely to deal with long-tail and occlusion concepts. Frequently co-occurred concepts in the same scene, e.g. bedroom and bed, can provide common-sense knowledge to discover other semantic-related concepts.  In this paper, we develop a Scene Concept Graph (SCG) by aggregating image scene graphs and extracting frequently co-occurred concept pairs as scene common-sense knowledge. Moreover, we propose a novel model to incorporate this knowledge to improve image-text matching. Specifically, semantic concepts are detected from images and then expanded by the SCG. After learning to select relevant contextual concepts, we fuse their representations with the image embedding feature to feed into the matching module. Extensive experiments are conducted on Flickr30K and MSCOCO datasets, and prove that our model achieves state-of-the-art results due to the effectiveness of incorporating the external SCG.","doi":"10.24963/ijcai.2019/720","url":"https://ijcai.org/proceedings/2019/720","pdf":"https://ijcai.org/proceedings/2019/0720.pdf","markdown":"https://ijcai.org/proceedings/2019/720.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/720","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":720,"pages":{"first":5182,"last":5189},"keywords":["Natural Language Processing: Information Retrieval","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[],"references":[{"position":1,"raw":"[Belilovsky et al., 2016] Eugene Belilovsky, Matthew Blaschko, Jamie Kiros, Raquel Urtasun, and Richard Zemel. Joint embeddings of scene graphs and images. In International Conference On Learning Representations- Workshop, 2016.","title":"Joint embeddings of scene graphs and images","authors":"Eugene Belilovsky, Matthew Blaschko, Jamie Kiros, Raquel Urtasun, Richard Zemel","year":"2016","venue":"International Conference On Learning Representations-Workshop","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Clevert et al., 2016] Djorkarne Clevert, Thomas Un- terthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). international conference on learning representations, 2016.","title":"Djorkarne Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units","authors":"Clevert","year":"2016","venue":"international conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierar- chical Image Database. In CVPR09, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"J Deng, W Dong, R Socher, L.-J Li, K Li, L Fei-Fei","year":"2009","venue":"CVPR09","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Faghri et al., 2017] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Im- proving visual-semantic embeddings with hard negatives. british machine vision conference, page 12, 2017.","title":"Vse++: Improving visual-semantic embeddings with hard negatives","authors":"Faghri","year":"2017","venue":"british machine vision conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Frome et al., 2013] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual-semantic embedding model. In Advances in neural information processing systems, pages 2121-2129, 2013.","title":"Devise: A deep visual-semantic embedding model","authors":"Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu et al., 2018] Jiuxiang Gu, Jianfei Cai, Shafiq Joty, Li Niu, and Gang Wang. Look, imagine and match: Im- proving textual-visual cross-modal retrieval with genera- tive models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7181- 7189, 2018.","title":"Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models","authors":"Gu","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2018] Yan Huang, Qi Wu, and Liang Wang. Learning semantic concepts and order for image and sen- tence matching. In Computer Vision and Pattern Recogni- tion, CVPR, 2018.","title":"Learning semantic concepts and order for image and sentence matching","authors":"Yan Huang, Qi Wu, Liang Wang","year":"2018","venue":"Computer Vision and Pattern Recognition, CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128- 3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kiela et al., 2018] Douwe Kiela, Alexis Conneau, Allan Jabri, and Maximilian Nickel. Learning visually grounded sentence representations. North American Chapter of the Association for Computational Linguistics, 1:408-418, 2018.","title":"Learning visually grounded sentence representations","authors":"Kiela","year":"2018","venue":"North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kiros et al., 2014] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014.","title":"Unifying visual-semantic embeddings with multimodal neural language models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard S Zemel","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Inter- national Journal of Computer Vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2018a] Chungwei Lee, Wei Fang, Chihkuan Yeh, and Yuchiang Frank Wang. Multi-label zero-shot learning with structured knowledge graphs. Computer Vision and Pattern Recognition, pages 1576-1585, 2018.","title":"Multi-label zero-shot learning with structured knowledge graphs","authors":"Lee","year":"2018","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2018b] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross atten- tion for image-text matching. In European Conference on Computer Vision (ECCV), 2018.","title":"Stacked cross attention for image-text matching","authors":"Lee","year":"2018","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin and Parikh, 2016] Xiao Lin and Devi Parikh. Leverag- ing visual question answering for image-caption ranking. In European Conference on Computer Vision, pages 261- 277. Springer, 2016.","title":"Leveraging visual question answering for image-caption ranking","authors":"Xiao Lin, Devi Parikh","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Marino et al., 2017] Kenneth Marino, Ruslan Salakhutdi- nov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. Computer Vi- sion and Pattern Recognition, pages 20-28, 2017.","title":"The more you know: Using knowledge graphs for image classification","authors":"Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta","year":"2017","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mogadala et al., 2017] Aditya Mogadala, Umanga Bista, Lexing Xie, and Achim Rettinger. Describing natural im- ages containing novel objects with knowledge guided as- sitance. arXiv preprint arXiv:1710.06303, 2017.","title":"Describing natural images containing novel objects with knowledge guided assitance","authors":"Aditya Mogadala, Umanga Bista, Lexing Xie, Achim Rettinger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.","title":"Facenet: A unified embedding for face recognition and clustering","authors":"Schroff","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2006] Huan Wang, Song Liu, and Liang-Tien Chia. Does ontology help in image retrieval?: a compar- ison between keyword, text ontology and multi-modality ontology approaches. In Proceedings of the 14th ACM international conference on Multimedia, pages 109-112. ACM, 2006.","title":"Does ontology help in image retrieval?: a comparison between keyword, text ontology and multi-modality ontology approaches","authors":"Huan Wang, Song Liu, Liang-Tien Chia","year":"2006","venue":"Proceedings of the 14th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5005- 5013, 2016.","title":"Learning deep structure-preserving image-text embeddings","authors":"Liwei Wang, Yin Li, Svetlana Lazebnik","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recog- nition by recurrently discovering attentional regions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 464-472, 2017.","title":"Multi-label image recognition by recurrently discovering attentional regions","authors":"Wang","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2018.","title":"Learning two-branch neural networks for image-text matching tasks","authors":"Liwei Wang, Yin Li, Jing Huang, Svetlana Lazebnik","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2016] Qi Wu, Peng Wang, Chunhua Shen, An- thony R Dick, and Anton Van Den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. Computer Vision and Pattern Recognition, pages 4622-4630, 2016.","title":"Ask me anything: Free-form visual question answering based on knowledge from external sources","authors":"Qi Wu, Peng Wang, Chunhua Shen, Anthony R Dick, Anton Van Den, Hengel","year":"2016","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2016] Liang Xie, Jialie Shen, Lei Zhu, et al. On- line cross-modal hashing for web image retrieval. In AAAI, pages 294-300, 2016.","title":"Online cross-modal hashing for web image retrieval","authors":"Liang Xie, Jialie Shen, Lei Zhu","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier","year":"2014","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2017] Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, and Yi-Dong Shen. Dual-path convolutional image-text embedding. arXiv preprint arXiv:1711.05535, 2017.","title":"Dual-path convolutional image-text embedding","authors":"Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Yi-Dong Shen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2018] Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Com- monsense knowledge aware conversation generation with graph attention. In IJCAI, pages 4623-4629, 2018.","title":"Commonsense knowledge aware conversation generation with graph attention","authors":"Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, Xiaoyan Zhu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/643"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}