{"type":"ScholarlyArticle","title":"Declaration-based Prompt Tuning for Visual Question Answering","authors":[{"name":"Yuhang Liu","slug":"yuhang-liu","given":"Yuhang","family":"Liu","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology , China; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL) , China"},{"name":"Wei Wei","slug":"wei-wei","given":"Wei","family":"Wei","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology , China; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL) , China"},{"name":"Daowan Peng","slug":"daowan-peng","given":"Daowan","family":"Peng","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology , China; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL) , China"},{"name":"Feida Zhu","slug":"feida-zhu","given":"Feida","family":"Zhu","affiliation":"School of Computing and Information Systems , Singapore Management University , Singapore"}],"abstract":"In recent years, the pre-training-then-fine-tuning paradigm has yielded immense success on a wide spectrum of cross-modal tasks, such as visual question answering (VQA), in which a visual-language (VL) model is first optimized via self-supervised task objectives, e.g., masked language modeling (MLM) and image-text matching (ITM), and then fine-tuned to adapt to downstream task (e.g., VQA) via a brand-new objective function, e.g., answer prediction. However, the inconsistency of the objective forms not only severely limits the generalization of pre-trained VL models to downstream tasks, but also requires a large amount of labeled data for fine-tuning. To alleviate the problem, we propose an innovative VL fine-tuning paradigm (named Declaration-based Prompt Tuning, abbreviated as DPT), which fine-tunes the model for downstream VQA using the pre-training objectives, boosting the effective adaptation of pre-trained\r\nmodels to the downstream task. Specifically, DPT reformulates the VQA task via (1) textual adaptation, which converts the given questions into declarative sentence form for prompt-tuning, and (2) task adaptation, which optimizes the objective function of VQA problem in the manner of pre-training phase. Experimental results on GQA dataset show that DPT outperforms the fine-tuned counterpart by a large margin regarding accuracy in both fully-supervised (2.68%) and zero-shot/fewshot (over 31%) settings. All the data and codes will be available to facilitate future research.","doi":"10.24963/ijcai.2022/453","url":"https://ijcai.org/proceedings/2022/453","pdf":"https://ijcai.org/proceedings/2022/0453.pdf","markdown":"https://ijcai.org/proceedings/2022/453.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/453","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":453,"pages":{"first":3264,"last":3270},"keywords":["Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Vision and language","Machine Learning: Multi-modal learning","Natural Language Processing: Question Answering"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986545","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984688","length_sec":299,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Agrawal et al., 2015] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. IJCV, pages 4-31, 2015.","title":"Vqa: Visual question answering","authors":"Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C Lawrence Zitnick, Devi Parikh, Dhruv Batra","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pages 104-120. Springer, 2020.","title":"Uniter: Universal image-text representation learning","authors":"Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu. Meta mod- ule network for compositional visual reasoning. In WACV, pages 655-664, 2021.","title":"Meta module network for compositional visual reasoning","authors":"Chen","year":"2021","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2020] Ling Cheng, Wei Wei, Xianling Mao, Yong Liu, and Chunyan Miao. Stack-vs: Stacked visual- semantic attention for image caption generation. IEEE Ac- cess, 8:154953-154965, 2020.","title":"Stack-vs: Stacked visualsemantic attention for image caption generation","authors":"Ling Cheng, Wei Wei, Xianling Mao, Yong Liu, Chunyan Miao","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2021] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text gen- eration. In ICML, pages 1931-1942, 2021.","title":"Unifying vision-and-language tasks via text generation","authors":"Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gan et al., 2020] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. NIPS, pages 6616-6628, 2020.","title":"Large-scale adversarial training for vision-and-language representation learning","authors":"Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, Jingjing Liu","year":"2020","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hudson and Manning, 2019a] Drew A Hudson and Christo- pher D Manning. Gqa: A new dataset for real-world vi- sual reasoning and compositional question answering. In CVPR, pages 6700-6709, 2019.","title":"Gqa: A new dataset for real-world visual reasoning and compositional question answering","authors":"Manning ; Hudson, A Drew, Christopher D Hudson, Manning","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hudson and Manning, 2019b] Drew A Hudson and Christo- pher D Manning. Learning by abstraction: the neural state machine. In NIPS, pages 5903-5916, 2019.","title":"Learning by abstraction: the neural state machine","authors":"Manning ; Pher D Hudson, Manning","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2021] Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters? low-resource prompt- based learning for vision-language models. arXiv preprint arXiv:2110.08484, 2021.","title":"A good prompt is worth millions of parameters? low-resource promptbased learning for vision-language models","authors":"Jin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kamath et al., 2021] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi- modal understanding. In ICCV, pages 1780-1790, 2021.","title":"Mdetr-modulated detection for end-to-end multimodal understanding","authors":"Aishwarya Kamath, Mannat Singh, Yann Lecun, Gabriel Synnaeve, Ishan Misra, Nicolas Carion","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2021] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without con- volution or region supervision. In ICML, pages 5583- 5594, 2021.","title":"Vilt: Vision-and-language transformer without convolution or region supervision","authors":"Kim","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pages 121-137, 2020.","title":"Oscar: Object-semantics aligned pre-training for vision-language tasks","authors":"Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021a] Daizong Liu, Shuangjie Xu, Xiao-Yang Liu, Zichuan Xu, Wei Wei, and Pan Zhou. Spatiotemporal graph neural network based mask reconstruction for video object segmentation. In AAAI, pages 2100-2108, 2021.","title":"Spatiotemporal graph neural network based mask reconstruction for video object segmentation","authors":"Liu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021b] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586, 2021.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: pretraining task-agnostic visiolinguis- tic representations for vision-and-language tasks. In NIPS, pages 13-23, 2019.","title":"Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Raffel et al., 2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Su et al., 2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530, 2019.","title":"Vl-bert: Pre-training of generic visual-linguistic representations","authors":"Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tan and Bansal, 2019] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP, pages 5100-5111, 2019.","title":"Lxmert: Learning cross-modality encoder representations from transformers","authors":"Hao Tan, Mohit Bansal","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tsimpoukelli et al., 2021] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NIPS, 34:200-212, 2021.","title":"Multimodal few-shot learning with frozen language models","authors":"Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, Oriol Eslami, Felix Vinyals, Hill","year":"2021","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yao et al., 2021] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Col- orful prompt tuning for pre-trained vision-language mod- els. arXiv preprint arXiv:2109.11797, 2021.","title":"Cpt: Colorful prompt tuning for pre-trained vision-language models","authors":"Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2021] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representa- tions in vision-language models. CVPR, pages 5575-5584, 2021.","title":"Vinvl: Revisiting visual representations in vision-language models","authors":"Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2021] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134, 2021.","title":"Learning to prompt for vision-language models","authors":"Zhou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}