{"type":"ScholarlyArticle","title":"Vision-and-Language Pretrained Models: A Survey","authors":[{"name":"Siqu Long","slug":"siqu-long","given":"Siqu","family":"Long","affiliation":"School of Computer Science , The University of Sydney , Australia"},{"name":"Feiqi Cao","slug":"feiqi-cao","given":"Feiqi","family":"Cao","affiliation":"School of Computer Science , The University of Sydney , Australia"},{"name":"Soyeon Caren Han","slug":"soyeon-caren-han","given":"Soyeon Caren","family":"Han","affiliation":"School of Computer Science , The University of Sydney , Australia"},{"name":"Haiqin Yang","slug":"haiqin-yang","given":"Haiqin","family":"Yang","affiliation":"International Digital Economy Academy (IDEA) , China"}],"abstract":"Pretrained models have produced great success in both Computer Vision (CV) and Natural Language Processing (NLP). This progress leads to learning joint representations of vision and language pretraining by feeding visual and linguistic contents into a multi-layer transformer, Visual-Language Pretrained Models (VLPMs). In this paper, we present an overview of the major advances achieved in VLPMs for producing joint representations of vision and language. As the preliminaries, we briefly describe the general task definition and genetic architecture of VLPMs. We first discuss the language and vision data encoding methods and then present the mainstream VLPM structure as the core content. We further summarise several essential pretraining and fine-tuning strategies. Finally, we highlight three future directions for both CV and NLP researchers to provide insightful guidance.","doi":"10.24963/ijcai.2022/773","url":"https://ijcai.org/proceedings/2022/773","pdf":"https://ijcai.org/proceedings/2022/0773.pdf","markdown":"https://ijcai.org/proceedings/2022/773.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/773","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Survey Track","track":"Survey Track","number":773,"pages":{"first":5530,"last":5537},"keywords":["Survey Track: Multidisciplinary Topics and Applications","Survey Track: Natural Language Processing","Survey Track: Computer Vision"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987297","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985440","length_sec":717,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bugliarello et al., 2021] Emanuele Bugliarello, Ryan Cot- terell, Naoaki Okazaki, and Desmond Elliott. Multi- modal pretraining unmasked: A meta-analysis and a uni- fied framework of vision-and-language berts. Transactions of the Association for Computational Linguistics, 9:978- 994, 2021.","title":"Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language berts","authors":"Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, Desmond Elliott","year":"2021","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representa- tion learning. In European conference on computer vision, pages 104-120. Springer, 2020.","title":"Uniter: Universal image-text representation learning","authors":"Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chiou et al., 2021] Meng-Jiun Chiou, Roger Zimmermann, and Jiashi Feng. Visual relationship detection with visual- linguistic knowledge from multimodal representations. IEEE Access, 9:50441-50451, 2021.","title":"Visual relationship detection with visuallinguistic knowledge from multimodal representations","authors":"Meng-Jiun Chiou, Roger Zimmermann, Jiashi Feng","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fei et al., 2021] Hongliang Fei, Tan Yu, and Ping Li. Cross- lingual cross-modal pretraining for multimodal retrieval. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies, 2021.","title":"Crosslingual cross-modal pretraining for multimodal retrieval","authors":"Hongliang Fei, Tan Yu, Ping Li","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gan et al., 2020] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. In NeurIPS, 2020.","title":"Large-scale adversarial training for vision-and-language representation learning","authors":"Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, Jingjing Liu","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2020] Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang. Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval. In Proceedings of the 43rd Inter- national ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, pages 2251-2260, 2020.","title":"Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval","authors":"Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, Hao Wang","year":"2020","venue":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hao et al., 2020] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-language navigation via pre- training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137- 13146, 2020.","title":"Towards learning a generic agent for vision-and-language navigation via pretraining","authors":"Hao","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2021] Yicong Hong, Qi Wu, Yuankai Qi, Cris- tian Rodriguez-Opazo, and Stephen Gould. Vln bert: A recurrent vision-and-language bert for navigation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1643-1653, 2021.","title":"Vln bert: A recurrent vision-and-language bert for navigation","authors":"Hong","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2021] Xiaowei Hu, Xi Yin, Kevin Lin, Lei Zhang, Jianfeng Gao, Lijuan Wang, and Zicheng Liu. Vivo: Vi- sual vocabulary pre-training for novel object captioning. Proceedings of the AAAI Conference on Artificial Intelli- gence, 35(2):1575-1583, May 2021.","title":"Vivo: Visual vocabulary pre-training for novel object captioning","authors":"Xiaowei Hu, Xi Yin, Kevin Lin, Lei Zhang, Jianfeng Gao, Lijuan Wang, Zicheng Liu","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2020] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020.","title":"Pixel-bert: Aligning image pixels with text by deep multi-modal transformers","authors":"Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2021] Zhicheng Huang, Zhaoyang Zeng, Yu- pan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. See- ing out of the box: End-to-end pre-training for vision- language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12976-12985, 2021.","title":"Seeing out of the box: End-to-end pre-training for visionlanguage representation learning","authors":"Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jia et al., 2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text su- pervision. arXiv preprint arXiv:2102.05918, 2021.","title":"Scaling up visual and vision-language representation learning with noisy text supervision","authors":"Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, V Quoc, Yunhsuan Le, Zhen Sung, Tom Li, Duerig","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2021] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer with- out convolution or region supervision. arXiv preprint arXiv:2102.03334, 2021.","title":"Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision","authors":"Kim","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: Asimple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.","title":"Visualbert: Asimple and performant baseline for vision and language","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2020a] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 11336-11344, 2020.","title":"Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training","authors":"Li","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2020b] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121-137. Springer, 2020.","title":"Oscar: Object-semantics aligned pre-training for vision-language tasks","authors":"Li","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021a] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and gener- ation via cross-modal contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Con- ference on Natural Language Processing (Volume 1: Long Papers), pages 2592-2607, 2021.","title":"Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning","authors":"Li","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2021b] Yehao Li, Yingwei Pan, Ting Yao, Jing- wen Chen, and Tao Mei. Scheduled sampling in vision- language pretraining with decoupled encoder-decoder net- work. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8518-8526, 2021.","title":"Scheduled sampling in visionlanguage pretraining with decoupled encoder-decoder network","authors":"Li","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. Ad- vances in NeurIPS, 32:13-23, 2019.","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lu et al., 2020] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437-10446, 2020.","title":"12-in-1: Multitask vision and language representation learning","authors":"Lu","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Majumdar et al., 2020] Arjun Majumdar, Ayush Shrivas- tava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-language navigation with image-text pairs from the web. In European Conference on Computer Vision, pages 259-274. Springer, 2020.","title":"Improving vision-and-language navigation with image-text pairs from the web","authors":"Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mogadala et al., 2021] Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. Journal of Artificial Intelligence Research, 71:1183-1317, 2021.","title":"Trends in integration of vision and language research: A survey of tasks, datasets, and methods","authors":"Aditya Mogadala, Marimuthu Kalimuthu, Dietrich Klakow","year":"2021","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Murahari et al., 2020] Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. In Eu- ropean Conference on Computer Vision, pages 336-352. Springer, 2020.","title":"Large-scale pretraining for visual dialog: A simple state-of-the-art baseline","authors":"Murahari","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ni et al., 2021] Minheng Ni, Haoyang Huang, Lin Su, Ed- ward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan. M3p: Learning universal representations via multitask multilingual multimodal pre-training. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3977-3986, 2021.","title":"M3p: Learning universal representations via multitask multilingual multimodal pre-training","authors":"Ni","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual mod- els from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Singh et al., 2021] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:2112.04482, 2021.","title":"Flava: A foundational language and vision alignment model","authors":"Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, Douwe Kiela","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Su et al., 2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In Interna- tional Conference on Learning Representations, 2019.","title":"Vl-bert: Pre-training of generic visual-linguistic representations","authors":"Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sun et al., 2021] Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Jingjing Liu. Light- ningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pages 982-997, 2021.","title":"Lightningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval","authors":"Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, Jingjing Liu","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tan and Bansal, 2019] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 5100-5111, 2019.","title":"Lxmert: Learning cross-modality encoder representations from transformers","authors":"Hao Tan, Mohit Bansal","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2020] Yue Wang, Shafiq Joty, Michael Lyu, Ir- win King, Caiming Xiong, and Steven CH Hoi. Vd-bert: A unified vision and dialog transformer with bert. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3325- 3338, 2020.","title":"Vd-bert: A unified vision and dialog transformer with bert","authors":"Yue Wang, Shafiq Joty, Michael Lyu, Irwin King, Caiming Xiong, Steven Ch Hoi","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2021a] Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre- training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358, 2021.","title":"Vlmo: Unified vision-language pretraining with mixture-of-modality-experts","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2021b] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervi- sion. arXiv preprint arXiv:2108.10904, 2021.","title":"Simvlm: Simple visual language model pretraining with weak supervision","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wen et al., 2021] Keyu Wen, Jin Xia, Yuanyuan Huang, Linyang Li, Jiayan Xu, and Jie Shao. Cookie: Con- trastive cross-modal knowledge sharing pre-training for vision-language representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2208-2217, 2021.","title":"Cookie: Contrastive cross-modal knowledge sharing pre-training for vision-language representation","authors":"Keyu Wen, Jin Xia, Yuanyuan Huang, Linyang Li, Jiayan Xu, Jie Shao","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xia et al., 2021] Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, and Ming Zhou. Xgpt: Cross-modal generative pre-training for image captioning. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 786-797. Springer, 2021.","title":"Xgpt: Cross-modal generative pre-training for image captioning","authors":"Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, Ming Zhou","year":"2021","venue":"CCF International Conference on Natural Language Processing and Chinese Computing","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang et al., 2021] Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text-caption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8751-8761, 2021.","title":"Tap: Text-aware pre-training for text-vqa and text-caption","authors":"Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, Jiebo Luo","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yu et al., 2021] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowl- edge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3208-3216, 2021.","title":"Ernie-vil: Knowledge enhanced vision-language representations through scene graphs","authors":"Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hua Hao Tian, Haifeng Wu, Wang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2021] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representa- tions in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.","title":"Vinvl: Revisiting visual representations in vision-language models","authors":"Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhou et al., 2020] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Uni- fied vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-13049, 2020.","title":"Unified vision-language pre-training for image captioning and vqa","authors":"Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, Jianfeng Gao","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhuge et al., 2021] Mingchen Zhuge, Dehong Gao, Deng- Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, and Ling Shao. Kaleido-bert: Vision-language pre-training on fashion domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12647-12657, 2021.","title":"Kaleido-bert: Vision-language pre-training on fashion domain","authors":"Mingchen Zhuge, Dehong Gao, Deng-Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, Ling Shao","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}