{"type":"ScholarlyArticle","title":"Black-box Prompt Tuning for Vision-Language Model as a Service","authors":[{"name":"Lang Yu","slug":"lang-yu","given":"Lang","family":"Yu","affiliation":"School of Computer Science and Technology , East China Normal University; Shanghai Institute of AI for Education , East China Normal University"},{"name":"Qin Chen","slug":"qin-chen","given":"Qin","family":"Chen","affiliation":"School of Computer Science and Technology , East China Normal University; Shanghai Institute of AI for Education , East China Normal University"},{"name":"Jiaju Lin","slug":"jiaju-lin","given":"Jiaju","family":"Lin","affiliation":"School of Computer Science and Technology , East China Normal University"},{"name":"Liang He","slug":"liang-he","given":"Liang","family":"He","affiliation":"School of Computer Science and Technology , East China Normal University; Shanghai Institute of AI for Education , East China Normal University"}],"abstract":"In the scenario of Model-as-a-Service (MaaS), pre-trained models are usually released as inference APIs. Users are allowed to query those models with manually crafted prompts. Without accessing the network structure and gradient information, it's tricky to perform continuous prompt tuning on MaaS, especially for vision-language models (VLMs) considering cross-modal interaction. In this paper, we propose a black-box prompt tuning framework for VLMs to learn task-relevant prompts without back-propagation. In particular, the vision and language prompts are jointly optimized in the intrinsic parameter subspace with various evolution strategies. Different prompt variants are also explored to enhance the cross-model interaction. Experimental results show that our proposed black-box prompt tuning framework outperforms both hand-crafted prompt engineering and gradient-based prompt learning methods, which serves as evidence of its capability to train task-relevant prompts in a derivative-free manner.","doi":"10.24963/ijcai.2023/187","url":"https://ijcai.org/proceedings/2023/187","pdf":"https://ijcai.org/proceedings/2023/0187.pdf","markdown":"https://ijcai.org/proceedings/2023/187.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/187","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":187,"pages":{"first":1686,"last":1694},"keywords":["Machine Learning: ML: Evolutionary learning","Computer Vision: CV: Vision and language","Machine Learning: ML: Multi-modal learning"],"videos":[],"references":[{"position":1,"raw":"[Aghajanyan et al., 2021] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 7319-7328, 2021.","title":"Intrinsic dimensionality explains the effectiveness of language model fine-tuning","authors":"Armen Aghajanyan, Sonal Gupta, Luke Zettlemoyer","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beyer and Sendhoff, 2017] Hans-Georg Beyer and Bern- hard Sendhoff. Simplify your covariance matrix adapta- tion evolution strategy. IEEE Transactions on Evolution- ary Computation, 21(5):746-759, 2017.","title":"Simplify your covariance matrix adaptation evolution strategy","authors":"Sendhoff ; Hans-Georg Beyer, Bernhard Beyer, Sendhoff","year":"2017","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bossard et al., 2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative com- ponents with random forests. In European conference on computer vision, pages 446-461. Springer, 2014.","title":"Food-101-mining discriminative components with random forests","authors":"Lukas Bossard, Matthieu Guillaumin, Luc Van Gool","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cimpoi et al., 2014] Mircea Cimpoi, Subhransu Maji, Ia- sonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3606-3613, 2014.","title":"Describing textures in the wild","authors":"Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, Andrea Vedaldi","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Du et al., 2022] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202.10936, 2022.","title":"A survey of vision-language pre-trained models","authors":"Yifan Du, Zikang Liu, Junyi Li, Wayne Xin Zhao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/762"},{"position":9,"raw":"[Fei-Fei et al., 2004] Li Fei-Fei, Rob Fergus, and Pietro Per- ona. Learning generative visual models from few train- ing examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.","title":"Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories","authors":"Li Fei-Fei, Rob Fergus, Pietro Perona","year":"2004","venue":"2004 conference on computer vision and pattern recognition workshop","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hansen and Ostermeier, 2001] Nikolaus Hansen and An- dreas Ostermeier. Completely derandomized self- adaptation in evolution strategies. Evolutionary compu- tation, 9(2):159-195, 2001.","title":"Completely derandomized selfadaptation in evolution strategies","authors":"Ostermeier Hansen","year":"2001","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2020] Xiaoyu He, Zibin Zheng, and Yuren Zhou. Mmes: Mixture model-based evolution strategy for large- scale optimization. IEEE Transactions on Evolutionary Computation, 25(2):320-333, 2020.","title":"Mmes: Mixture model-based evolution strategy for largescale optimization","authors":"Xiaoyu He, Zibin Zheng, Yuren Zhou","year":"2020","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Helber et al., 2019] Patrick Helber, Benjamin Bischke, An- dreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Top- ics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.","title":"Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification","authors":"Patrick Helber, Benjamin Bischke, Andreas Dengel, Damian Borth","year":"2019","venue":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Houlsby et al., 2019] Neil Houlsby, Andrei Giurgiu, Stanis- law Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Inter- national Conference on Machine Learning, pages 2790- 2799. PMLR, 2019.","title":"Parameter-efficient transfer learning for nlp","authors":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jia et al., 2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), 2022.","title":"Visual prompt tuning","authors":"Jia","year":"2022","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Larson et al., 2019] Jeffrey Larson, Matt Menickelly, and Stefan M Wild. Derivative-free optimization methods. Acta Numerica, 28:287-404, 2019.","title":"Derivative-free optimization methods","authors":"Jeffrey Larson, Matt Menickelly, Stefan M Wild","year":"2019","venue":"Acta Numerica","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics.","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Lisa Xiang, Percy Li, Liang","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2022] Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu. Late prompt tuning: A late prompt could be better than many prompts. arXiv preprint arXiv:2210.11292, 2022.","title":"Late prompt tuning: A late prompt could be better than many prompts","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Loshchilov et al., 2018] Ilya Loshchilov, Tobias Glasmach- ers, and Hans-Georg Beyer. Large scale black-box op- timization by limited-memory matrix adaptation. IEEE Transactions on Evolutionary Computation, 23(2):353- 358, 2018.","title":"Large scale black-box optimization by limited-memory matrix adaptation","authors":"Loshchilov","year":"2018","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Loshchilov, 2014] Ilya Loshchilov. A computationally effi- cient limited memory cma-es for large scale optimization. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pages 397-404, 2014.","title":"Ilya Loshchilov. A computationally efficient limited memory cma-es for large scale optimization","authors":"Loshchilov","year":"2014","venue":"Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nilsback and Zisserman, 2008] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Confer- ence on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.","title":"Automated flower classification over a large number of classes","authors":"Maria-Elena Nilsback, Andrew Zisserman","year":"2008","venue":"Sixth Indian Conference on Computer Vision, Graphics & Image Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Parkhi et al., 2012] Omkar M Parkhi, Andrea Vedaldi, An- drew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recogni- tion, pages 3498-3505. IEEE, 2012.","title":"Cats and dogs","authors":"Andrea Omkar M Parkhi, Andrew Vedaldi, Zisserman, Jawahar","year":"2012","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shahriari et al., 2015] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimiza- tion. Proceedings of the IEEE, 104(1):148-175, 2015.","title":"Taking the human out of the loop: A review of bayesian optimization","authors":"Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, Nando De Freitas","year":"2015","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2022a] Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. Bbtv2: Towards a gradient-free future with large language models. In Proceedings of EMNLP, 2022.","title":"Bbtv2: Towards a gradient-free future with large language models","authors":"Sun","year":"2022","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2022b] Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service. In Proceedings of ICML, 2022.","title":"Black-box tuning for language-model-as-a-service","authors":"Sun","year":"2022","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tian et al., 2020] Yonglong Tian, Yue Wang, Dilip Krish- nan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, pages 266-282. Springer, 2020.","title":"Rethinking few-shot image classification: a good embedding is all you need?","authors":"Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, Phillip Isola","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xiao et al., 2010] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.","title":"Sun database: Large-scale scene recognition from abbey to zoo","authors":"Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, Antonio Torralba","year":"2010","venue":"IEEE computer society conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.","title":"Learning to prompt for vision-language models","authors":"Zhou","year":"2022","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}