{"type":"ScholarlyArticle","title":"C3L: Content Correlated Vision-Language Instruction Tuning Data Generation via Contrastive Learning","authors":[{"name":"Ji Ma","slug":"ji-ma","given":"Ji","family":"Ma","affiliation":"School of Computer Science , Northwestern Polytechnical University , China; Ningbo Institute , Northwestern Polytechnical University , China; National Engineering Laboratory for Integrated Aero -Space-Ground-Ocean , China"},{"name":"Wei Suo","slug":"wei-suo","given":"Wei","family":"Suo","affiliation":"School of Computer Science , Northwestern Polytechnical University , China; Ningbo Institute , Northwestern Polytechnical University , China; National Engineering Laboratory for Integrated Aero -Space-Ground-Ocean , China"},{"name":"Peng Wang","slug":"peng-wang","given":"Peng","family":"Wang","affiliation":"School of Computer Science , Northwestern Polytechnical University , China; Ningbo Institute , Northwestern Polytechnical University , China; National Engineering Laboratory for Integrated Aero -Space-Ground-Ocean , China"},{"name":"Yanning Zhang","slug":"yanning-zhang","given":"Yanning","family":"Zhang","affiliation":"School of Computer Science , Northwestern Polytechnical University , China; Ningbo Institute , Northwestern Polytechnical University , China; National Engineering Laboratory for Integrated Aero -Space-Ground-Ocean , China"}],"abstract":"Vision-Language Instruction Tuning (VLIT) is a critical training phase for Large Vision-Language Models (LVLMs). With the improving capabilities of open-source LVLMs, researchers have increasingly turned to generate VLIT data by using open-source LVLMs and achieved significant progress. However, such data generation approaches are bottlenecked by the following challenges: 1) Since multi-modal models tend to be influenced by prior language knowledge, directly using LVLMs to generate VLIT data would inevitably lead to low content relevance between generated data and images. 2) To improve the ability of the models to generate VLIT data, previous methods have incorporated an additional training phase to boost the generative capacity. This process hurts the generalization of the models to unseen inputs (i.e., “exposure bias” problem). In this paper, we propose a new Content Correlated VLIT data generation  via Contrastive Learning (C3L). Specifically, we design a new content relevance module which enhances the content relevance between VLIT data and images by computing Image Instruction Correspondence Scores S(I2C). Moreover, a contrastive learning module is introduced to further boost the VLIT data generation capability of the LVLMs. A large number of automatic measures  on four benchmarks show the effectiveness of our method.","doi":"10.24963/ijcai.2024/128","url":"https://ijcai.org/proceedings/2024/128","pdf":"https://ijcai.org/proceedings/2024/0128.pdf","markdown":"https://ijcai.org/proceedings/2024/128.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/128","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":128,"pages":{"first":1155,"last":1163},"keywords":["Computer Vision: CV: Multimodal learning","Computer Vision: CV: Vision, language and reasoning","Natural Language Processing: NLP: Language models"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shi- jie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.","title":"Qwen-vl: A frontier large visionlanguage model with versatile abilities","authors":"Bai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bu et al., 2018] Yuheng Bu, Shaofeng Zou, Yingbin Liang, and Venugopal V Veeravalli. Estimation of kl divergence: Optimal minimax rate. IEEE Transactions on Information Theory, 64(4):2648-2674, 2018.","title":"Estimation of kl divergence: Optimal minimax rate","authors":"Yuheng Bu, Shaofeng Zou, Yingbin Liang, Venugopal, Veeravalli","year":"2018","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Changpinyo et al., 2022] Soravit Changpinyo, Doron Kuk- liansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Sori- cut. All you may need for vqa are image captions, 2022.","title":"Soravit Changpinyo","authors":"Changpinyo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, vol- ume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chowdhery et al., 2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling lan- guage modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.","title":"Palm: Scaling language modeling with pathways","authors":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chung et al., 2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.","title":"Scaling instruction-finetuned language models","authors":"Chung Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dai and Lin, 2017] Bo Dai and Dahua Lin. Contrastive learning for image captioning. Advances in Neural Infor- mation Processing Systems, 30, 2017.","title":"Contrastive learning for image captioning","authors":"Lin ; Dai, Bo Dai, Dahua Lin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2023] Wenliang Dai, Junnan Li, Dongxu Li, An- thony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with in- struction tuning, 2023.","title":"Instructblip: Towards general-purpose vision-language models with instruction tuning","authors":"Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng, Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Du et al., 2022] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: Gen- eral language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335, 2022.","title":"Glm: General language model pretraining with autoregressive blank infilling","authors":"Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Du et al., 2023a] Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning. arXiv preprint arXiv:2311.01487, 2023.","title":"What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning","authors":"Du","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Du et al., 2023b] Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning, 2023.","title":"What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning","authors":"Du","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gan et al., 2022] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends® in Computer Graphics and Vi- sion, 14(3-4):163-352, 2022.","title":"Vision-language pre-training: Basics, recent advances, and future trends","authors":"Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao","year":"2022","venue":"Foundations and Trends® in Computer Graphics and Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understand- ing in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 6904-6913, 2017.","title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gu et al., 2023] Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation mod- els. arXiv preprint arXiv:2307.12980, 2023.","title":"A systematic survey of prompt engineering on vision-language foundation models","authors":"Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Philip Volker Tresp, Torr","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Guan et al., 2023] Tianrui Guan, Fuxiao Liu, Xiyang Wu Ruiqi Xian Zongxia Li, Xiaoyu Liu Xijun Wang, Lichang Chen Furong Huang Yaser Yacoob, and Dinesh Manocha Tianyi Zhou. Hallusionbench: An advanced di- agnostic suite for entangled language hallucination & vi- sual illusion in large vision-language models. arXiv e- prints, pages arXiv-2310, 2023.","title":"Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models","authors":"Fuxiao Tianrui Guan, Xiyang Liu, Ruiqi Wu, Zongxia Xian, Xiaoyu Li, Liu Xijun, Lichang Wang, Furong Chen, Yaser Huang, Dinesh Yacoob, Tianyi Manocha, Zhou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hadsell et al., 2006] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), vol- ume 2, pages 1735-1742. IEEE, 2006.","title":"Dimensionality reduction by learning an invariant mapping","authors":"Raia Hadsell, Sumit Chopra, Yann Lecun","year":"2006","venue":"IEEE computer society conference on computer vision and pattern recognition (CVPR'06)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hamadi, 2023] Raby Hamadi. Large language models meet computer vision: A brief survey. arXiv preprint arXiv:2311.16673, 2023.","title":"Large language models meet computer vision: A brief survey","authors":"Raby Hamadi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lai et al., 2023] Chengen Lai, Shengli Song, Shiqi Meng, Jingyang Li, Sitong Yan, and Guangneng Hu. To- wards more faithful natural language explanation using multi-level contrastive learning in vqa. arXiv preprint arXiv:2312.13594, 2023.","title":"Towards more faithful natural language explanation using multi-level contrastive learning in vqa","authors":"Lai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Laurenc ¸on et al., 2023] Hugo Laurenc ¸on, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexan- der M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of in- terleaved image-text documents, 2023.","title":"Obelics: An open web-scale filtered dataset of interleaved image-text documents","authors":"Laurenc ¸on","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lee et al., 2020] Seanie Lee, Dong Bok Lee, and Sung Ju Hwang. Contrastive learning with adversarial perturba- tions for conditional text generation. In International Con- ference on Learning Representations, 2020.","title":"Contrastive learning with adversarial perturbations for conditional text generation","authors":"Seanie Lee, Dong Bok Lee, Sung Ju Hwang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2023a] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative compre- hension. arXiv preprint arXiv:2307.16125, 2023.","title":"Seed-bench: Benchmarking multimodal llms with generative comprehension","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2023b] Chen Li, Yixiao Ge, Dian Li, and Ying Shan. Vision-language instruction tuning: A review and analysis. arXiv preprint arXiv:2311.08172, 2023.","title":"Vision-language instruction tuning: A review and analysis","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2023c] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2023d] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating ob- ject hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023.","title":"Evaluating object hallucination in large vision-language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liang et al., 2020] Zujie Liang, Weitao Jiang, Haifeng Hu, and Jiaying Zhu. Learning to contrast the counterfactual samples for robust visual question answering. In Pro- ceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 3285-3292, 2020.","title":"Learning to contrast the counterfactual samples for robust visual question answering","authors":"Liang","year":"2020","venue":"Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Liu et al., 2023a] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallu- cination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565, 1, 2023.","title":"Mitigating hallucination in large multi-modal models via robust instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Liu et al., 2023b] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruc- tion tuning, 2023.","title":"Improved baselines with visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Liu et al., 2023c] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.","title":"Visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Liu et al., 2023d] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mm- bench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023.","title":"Mmbench: Is your multi-modal model an all-around player","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[OpenAI, 2023] R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.","title":"Gpt-4 technical report","authors":"Openai","year":"2023","venue":"View in Article","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Radford","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485- 5551, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Ranzato et al., 2015] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732, 2015.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Suo et al., 2023] Wei Suo, Mengyang Sun, Weisong Liu, Yiqi Gao, Peng Wang, Yanning Zhang, and Qi Wu. S3c: Semi-supervised vqa natural language explanation via self-critical learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2646-2656, 2023.","title":"S3c: Semi-supervised vqa natural language explanation via self-critical learning","authors":"Wei Suo, Mengyang Sun, Weisong Liu, Yiqi Gao, Peng Wang, Yanning Zhang, Qi Wu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard","authors":"Touvron","year":"2023","venue":"Llama: Open and efficient foundation language models","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2022] Yizhong Wang, Yeganeh Kordi, Swa- roop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.","title":"Self-instruct: Aligning language model with self generated instructions","authors":"Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, Hannaneh Hajishirzi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wang et al., 2023a] Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. Vigc: Visual instruction gen- eration and correction. arXiv preprint arXiv:2308.12714, 2023.","title":"Vigc: Visual instruction generation and correction","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wang et al., 2023b] Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023.","title":"To see is to believe: Prompting gpt-4v for better visual instruction tuning","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592, 2023.","title":"Minigpt-4: Enhancing vision-language understanding with advanced large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}