{"type":"ScholarlyArticle","title":"A Survey of Vision-Language Pre-Trained Models","authors":[{"name":"Yifan Du","slug":"yifan-du","given":"Yifan","family":"Du","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China; Beijing Key Laboratory of Big Data Management and Analysis Methods"},{"name":"Zikang Liu","slug":"zikang-liu","given":"Zikang","family":"Liu","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China"},{"name":"Junyi Li","slug":"junyi-li","given":"Junyi","family":"Li","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China; DIRO , Université de Montréal"},{"name":"Wayne Xin Zhao","slug":"wayne-xin-zhao","given":"Wayne Xin","family":"Zhao","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China; Beijing Key Laboratory of Big Data Management and Analysis Methods"}],"abstract":"As transformer evolves, pre-trained models have advanced at a breakneck pace in recent years. They have dominated the mainstream techniques in natural language processing (NLP) and computer vision (CV). How to adapt pre-training to the field of Vision-and-Language (V-L) learning and improve downstream task performance becomes a focus of multimodal learning. In this paper, we review the recent progress in Vision-Language Pre-Trained Models (VL-PTMs). As the core content, we first briefly introduce several ways to encode raw images and texts to single-modal embeddings before pre-training. Then, we dive into the mainstream architectures of VL-PTMs in modeling the interaction between text and image representations. We further present widely-used pre-training tasks, and then we introduce some common downstream tasks. We finally conclude this paper and present some promising research directions. Our survey aims to provide researchers with synthesis and pointer to related research.","doi":"10.24963/ijcai.2022/762","url":"https://ijcai.org/proceedings/2022/762","pdf":"https://ijcai.org/proceedings/2022/0762.pdf","markdown":"https://ijcai.org/proceedings/2022/762.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/762","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Survey Track","track":"Survey Track","number":762,"pages":{"first":5436,"last":5443},"keywords":["Survey Track: Computer Vision","Survey Track: Natural Language Processing"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987311","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985454","length_sec":649,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arevalo et al., 2017] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992, 2017.","title":"Gated multimodal units for information fusion","authors":"John Arevalo, Thamar Solorio, Manuel Montes-Y Gómez, Fabio A González","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baevski et al., 2022] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. URL https://ai. facebook. com/research/data2veca-general-framework- for-self-supervi sed-learning-in-speech-vision-and-la nguage/. Accessed, 2022.","title":"Data2vec: A general framework for self-supervised learning in speech, vision and language","authors":"Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli","year":"2022","venue":null,"doi":null,"url":"https://ai.facebook.com/research/data2veca-general-framework-for-self-supervised-learning-in-speech-vision-and-language/","internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, pages 1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pages 104-120. Springer, 2020.","title":"Uniter: Universal image-text representation learning","authors":"Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2020] Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, and Aniruddha Kembhavi. X- lxmert: Paint, caption and answer questions with multi- modal transformers. arXiv preprint arXiv:2009.11278, 2020.","title":"Xlxmert: Paint, caption and answer questions with multimodal transformers","authors":"Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cho et al., 2021] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text gen- eration. arXiv preprint arXiv:2102.02779, 2021.","title":"Unifying vision-and-language tasks via text generation","authors":"Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Desai et al., 2021] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image- text data created by the people, for the people. arXiv preprint arXiv:2111.11431, 2021.","title":"Redcaps: Web-curated imagetext data created by the people, for the people","authors":"Karan Desai, Gaurav Kaul, Zubin Aysola, Justin Johnson","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fang et al., 2021] Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Com- pressing visual-linguistic model via knowledge distilla- tion. In CVPR, pages 1428-1438, 2021.","title":"Compressing visual-linguistic model via knowledge distillation","authors":"Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, Zicheng Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fei et al., 2021] Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guox- ing Yang, Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang, et al. Wenlan 2.0: Make ai imagine via a multimodal foundation model. arXiv preprint arXiv:2110.14378, 2021.","title":"Wenlan 2.0: Make ai imagine via a multimodal foundation model","authors":"Fei","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gan et al., 2020] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. NeurIPS, pages 6616-6628, 2020.","title":"Large-scale adversarial training for vision-and-language representation learning","authors":"Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, Jingjing Liu","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2020] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020.","title":"Pixel-bert: Aligning image pixels with text by deep multi-modal transformers","authors":"Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2021] Zhicheng Huang, Zhaoyang Zeng, Yu- pan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. See- ing out of the box: End-to-end pre-training for vision- language representation learning. In CVPR, pages 12976- 12985, 2021.","title":"Seeing out of the box: End-to-end pre-training for visionlanguage representation learning","authors":"Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huo et al., 2021] Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. Wenlan: Bridging vision and language by large-scale multi-modal pre-training. arXiv preprint arXiv:2103.06561, 2021.","title":"Wenlan: Bridging vision and language by large-scale multi-modal pre-training","authors":"Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jia et al., 2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text su- pervision. arXiv preprint arXiv:2102.05918, 2021.","title":"Scaling up visual and vision-language representation learning with noisy text supervision","authors":"Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, V Quoc, Yunhsuan Le, Zhen Sung, Tom Li, Duerig","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kim et al., 2021] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer with- out convolution or region supervision. arXiv preprint arXiv:2102.03334, 2021.","title":"Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision","authors":"Kim","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, pages 32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lee et al., 2018] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pages 201-216, 2018.","title":"Stacked cross attention for image-text matching","authors":"Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, Xiaodong He","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.","title":"Visualbert: A simple and performant baseline for vision and language","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2020a] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, pages 11336-11344, 2020.","title":"Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training","authors":"Li","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2020b] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and gener- ation via cross-modal contrastive learning. arXiv preprint arXiv:2012.15409, 2020.","title":"Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning","authors":"Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2020c] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pages 121-137. Springer, 2020.","title":"Oscar: Object-semantics aligned pre-training for vision-language tasks","authors":"Li","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2021a] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.","title":"Align before fuse: Vision and language representation learning with momentum distillation","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2021b] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data ef- ficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021.","title":"Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lin et al., 2020] Junyang Lin, An Yang, Yichang Zhang, Jie Liu, Jingren Zhou, and Hongxia Yang. Interbert: Vision-and-language interaction for multi-modal pretrain- ing. arXiv preprint arXiv:2003.13198, 2020.","title":"Interbert: Vision-and-language interaction for multi-modal pretraining","authors":"Lin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019.","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Mogadala et al., 2021] Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. JAIR, pages 1183-1317, 2021.","title":"Trends in integration of vision and language research: A survey of tasks, datasets, and methods","authors":"Aditya Mogadala, Marimuthu Kalimuthu, Dietrich Klakow","year":"2021","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ordonez et al., 2011] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. NeurIPS, 2011.","title":"Im2text: Describing images using 1 million captioned photographs","authors":"Vicente Ordonez, Girish Kulkarni, Tamara Berg","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual mod- els from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ramesh et al., 2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pages 8821-8831, 2021.","title":"Zero-shot text-to-image generation","authors":"Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ruan and Jin, 2022] Ludan Ruan and Qin Jin. Survey: Transformer based video-language pre-training. AI Open, 2022.","title":"Survey: Transformer based video-language pre-training","authors":"Jin ; Ruan, Ludan Ruan, Qin Jin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Sharma et al., 2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pages 2556-2565, 2018.","title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning","authors":"Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition","authors":"Zisserman Simonyan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Singh et al., 2021] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:2112.04482, 2021.","title":"Flava: A foundational language and vision alignment model","authors":"Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, Douwe Kiela","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Su et al., 2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530, 2019.","title":"Vl-bert: Pre-training of generic visual-linguistic representations","authors":"Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Suhr et al., 2018] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for rea- soning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491, 2018.","title":"A corpus for reasoning about natural language grounded in photographs","authors":"Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, Yoav Artzi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Tan and Bansal, 2019] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019.","title":"Lxmert: Learning cross-modality encoder representations from transformers","authors":"Hao Tan, Mohit Bansal","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wang et al., 2021a] Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre- training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358, 2021.","title":"Vlmo: Unified vision-language pretraining with mixture-of-modality-experts","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wang et al., 2021b] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervi- sion. arXiv preprint arXiv:2108.10904, 2021.","title":"Simvlm: Simple visual language model pretraining with weak supervision","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Xia et al., 2021] Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, and Ming Zhou. Xgpt: Cross-modal generative pre-training for image captioning. In NLPCC, pages 786- 797, 2021.","title":"Xgpt: Cross-modal generative pre-training for image captioning","authors":"Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, Ming Zhou","year":"2021","venue":"NLPCC","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Xue et al., 2021] Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training. NeurIPS, 2021.","title":"Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training","authors":"Yupan Hongwei Xue, Bei Huang, Houwen Liu, Jianlong Peng, Houqiang Fu, Jiebo Li, Luo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Yang et al., 2022] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre- training with triple contrastive learning. arXiv preprint arXiv:2202.10401, 2022.","title":"Vision-language pretraining with triple contrastive learning","authors":"Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, Junzhou Huang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zellers et al., 2019] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, pages 6720- 6731, 2019.","title":"From recognition to cognition: Visual commonsense reasoning","authors":"Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yejin Choi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhou et al., 2020] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Uni- fied vision-language pre-training for image captioning and vqa. In AAAI, pages 13041-13049, 2020.","title":"Unified vision-language pre-training for image captioning and vqa","authors":"Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, Jianfeng Gao","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}