{"type":"ScholarlyArticle","title":"Multi-View Visual Semantic Embedding","authors":[{"name":"Zheng Li","slug":"zheng-li","given":"Zheng","family":"Li","affiliation":"Beijing Key Laboratory of Network System Architecture and Convergence , Beijing University of Posts and Telecommunications"},{"name":"Caili Guo","slug":"caili-guo","given":"Caili","family":"Guo","affiliation":"Beijing Key Laboratory of Network System Architecture and Convergence , Beijing University of Posts and Telecommunications; Beijing Laboratory of Advanced Information Networks , Beijing University of Posts and Telecommunications"},{"name":"Zerun Feng","slug":"zerun-feng","given":"Zerun","family":"Feng","affiliation":"Beijing Key Laboratory of Network System Architecture and Convergence , Beijing University of Posts and Telecommunications"},{"name":"Jenq-Neng Hwang","slug":"jenq-neng-hwang","given":"Jenq-Neng","family":"Hwang","affiliation":"University of Washington"},{"name":"Xijun Xue","slug":"xijun-xue","given":"Xijun","family":"Xue","affiliation":"China Telecom System Integration Co. , Ltd"}],"abstract":"Visual Semantic Embedding (VSE) is a dominant method for cross-modal vision-language retrieval. Its purpose is to learn an embedding space so that visual data can be embedded in a position close to the corresponding text description. However, there are large intra-class variations in the vision-language data. For example, multiple texts describing the same image may be described from different views, and the descriptions of different views are often dissimilar. The mainstream VSE method embeds samples from the same class in similar positions, which will suppress intra-class variations and lead to inferior generalization performance. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework, which learns multiple embeddings for one visual data and explicitly models intra-class variations. To optimize MV-VSE, a multi-view upper bound loss is proposed, and the multi-view embeddings are jointly optimized while retaining intra-class variations. MV-VSE is plug-and-play and can be applied to various VSE models and loss functions without excessively increasing model complexity. Experimental results on the Flickr30K and MS-COCO datasets demonstrate the superior performance of our framework.","doi":"10.24963/ijcai.2022/158","url":"https://ijcai.org/proceedings/2022/158","pdf":"https://ijcai.org/proceedings/2022/0158.pdf","markdown":"https://ijcai.org/proceedings/2022/158.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/158","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":158,"pages":{"first":1130,"last":1136},"keywords":["Computer Vision: Vision and language","Machine Learning: Multi-modal learning","Computer Vision: Image and Video retrieval"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987006","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985149","length_sec":643,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.","title":"Uniter: Universal image-text representation learning","authors":"Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pool- ing strategy for visual semantic embedding. In CVPR, 2021.","title":"Learning the best pooling strategy for visual semantic embedding","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Faghri et al., 2018] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Im- proving visual-semantic embeddings with hard negatives. In BMVC, 2018.","title":"Vse++: Improving visual-semantic embeddings with hard negatives","authors":"Faghri","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng et al., 2020] Zerun Feng, Zhimin Zeng, Caili Guo, and Zheng Li. Exploiting visual semantic reasoning for video- text retrieval. In IJCAI, 2020.","title":"Exploiting visual semantic reasoning for videotext retrieval","authors":"Feng","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Frome et al., 2013] Andrea Frome, Greg S Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Mar- c'Aurelio Ranzato, and Tomas Mikolov. Devise: a deep visual-semantic embedding model. In NeurIPS, 2013.","title":"Devise: a deep visual-semantic embedding model","authors":"Andrea Frome, Greg S Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'aurelio Ranzato, Tomas Mikolov","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2018] Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. Learning semantic concepts and order for image and sentence matching. In CVPR, 2018.","title":"Learning semantic concepts and order for image and sentence matching","authors":"Yan Huang, Qi Wu, Chunfeng Song, Liang Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2021] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without con- volution or region supervision. In ICML, 2021.","title":"Vilt: Vision-and-language transformer without convolution or region supervision","authors":"Kim","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2018] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, 2018.","title":"Stacked cross attention for image-text matching","authors":"Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, Xiaodong He","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In ICCV, 2019.","title":"Visual semantic reasoning for image-text matching","authors":"Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, Yun Fu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020] Zheng Li, Caili Guo, Bo Yang, Zerun Feng, and Hao Zhang. A novel convolutional architecture for video-text retrieval. In ICME, 2020.","title":"A novel convolutional architecture for video-text retrieval","authors":"Zheng Li, Caili Guo, Bo Yang, Zerun Feng, Hao Zhang","year":"2020","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Lin","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. In NeurIPS, 2019.","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qu et al., 2020] Leigang Qu, Meng Liu, Da Cao, Liqiang Nie, and Qi Tian. Context-aware multi-view summariza- tion network for image-text matching. In MM, 2020.","title":"Context-aware multi-view summarization network for image-text matching","authors":"Qu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In NuerIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NuerIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sanakoyeu et al., 2019] Artsiom Sanakoyeu, Vadim Tsch- ernezki, Uta Buchler, and Bjorn Ommer. Divide and con- quer the embedding space for metric learning. In CVPR, 2019.","title":"Divide and conquer the embedding space for metric learning","authors":"Artsiom Sanakoyeu, Vadim Tschernezki, Uta Buchler, Bjorn Ommer","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020] Haoran Wang, Ying Zhang, Zhong Ji, Yanwei Pang, and Lin Ma. Consensus-aware visual- semantic embedding for image-text matching. In ECCV, 2020.","title":"Consensus-aware visualsemantic embedding for image-text matching","authors":"Haoran Wang, Ying Zhang, Zhong Ji, Yanwei Pang, Lin Ma","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wehrmann et al., 2019] Jonatas Wehrmann, Douglas M Souza, Mauricio A Lopes, and Rodrigo C Barros. Language-agnostic visual-semantic embeddings. In ICCV, 2019.","title":"Language-agnostic visual-semantic embeddings","authors":"Jonatas Wehrmann, Douglas M Souza, Mauricio A Lopes, Rodrigo C Barros","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xuan et al., 2020] Hong Xuan, Abby Stylianou, and Robert Pless. Improved embeddings with easy positive triplet mining. In WACV, 2020.","title":"Improved embeddings with easy positive triplet mining","authors":"Xuan","year":"2020","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier","year":"2014","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhai and Wu, 2019] Andrew Zhai and Hao-Yu Wu. Classi- fication is a strong baseline for deep metric learning. In BMVC, 2019.","title":"Classification is a strong baseline for deep metric learning","authors":"Andrew Zhai, Hao-Yu Wu","year":"2019","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2022] Huatian Zhang, Zhendong Mao, Kun Zhang, and Yongdong Zhang. Show your faith: Cross- modal confidence-aware network for image-text matching. In AAAI, 2022.","title":"Show your faith: Crossmodal confidence-aware network for image-text matching","authors":"Huatian Zhang, Zhendong Mao, Kun Zhang, Yongdong Zhang","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zheng et al., 2021a] Wenzhao Zheng, Chengkun Wang, Ji- wen Lu, and Jie Zhou. Deep compositional metric learn- ing. In CVPR, 2021.","title":"Deep compositional metric learning","authors":"Zheng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2021b] Wenzhao Zheng, Borui Zhang, Jiwen Lu, and Jie Zhou. Deep relational metric learning. In IC- CV, 2021.","title":"Deep relational metric learning","authors":"Zheng","year":"2021","venue":"IC-CV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}