{"type":"ScholarlyArticle","title":"MSCI: Addressing CLIP's Inherent Limitations for Compositional Zero-Shot Learning","authors":[{"name":"Yue Wang","slug":"yue-wang","given":"Yue","family":"Wang","affiliation":"Nanjing University of Aeronautics and Astronautics , Nanjing , China"},{"name":"Shuai Xu","slug":"shuai-xu","given":"Shuai","family":"Xu","affiliation":"Nanjing University of Aeronautics and Astronautics , Nanjing , China; Key Laboratory of Social Computing and Cognitive Intelligence ( Dalian University of Technology) ,"},{"name":"Xuelin Zhu","slug":"xuelin-zhu","given":"Xuelin","family":"Zhu","affiliation":"The Hong Kong Polytechnic University , Hong Kong , China; Ministry of Education , China"},{"name":"Yicong Li","slug":"yicong-li","given":"Yicong","family":"Li","affiliation":"Nanjing University of Aeronautics and Astronautics , Nanjing , China"}],"abstract":"Compositional Zero-Shot Learning (CZSL) aims to recognize unseen state-object combinations by leveraging known combinations. Existing studies basically rely on the cross-modal alignment capabilities of CLIP but tend to overlook its limitations in capturing fine-grained local features, which arise from its architectural and training paradigm. To address this issue, we propose a Multi-Stage Cross-modal Interaction (MSCI) model that effectively explores and utilizes intermediate-layer information from CLIP's visual encoder. Specifically, we design two self-adaptive aggregators to extract local information from low-level visual features and integrate global information from high-level visual features, respectively. These key information are progressively incorporated into textual representations through a stage-by-stage interaction mechanism, significantly enhancing the model’s perception capability for fine-grained local visual information. Additionally, MSCI dynamically adjusts the attention weights between global and local visual information based on different combinations, as well as different elements within the same combination, allowing it to flexibly adapt to diverse scenarios. Experiments on three widely used datasets fully validate the effectiveness and superiority of the proposed model. Data and code are available at https://github.com/ltpwy/MSCI.","doi":"10.24963/ijcai.2025/224","url":"https://ijcai.org/proceedings/2025/224","pdf":"https://ijcai.org/proceedings/2025/0224.pdf","markdown":"https://ijcai.org/proceedings/2025/224.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/224","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Computer Vision","track":"Main Track","number":224,"pages":{"first":2009,"last":2017},"keywords":["Computer Vision: CV: Structural and model-based approaches, knowledge representation and reasoning","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Machine Learning: ML: Classification","Machine Learning: ML: Deep learning architectures"],"videos":[],"references":[{"position":1,"raw":"[Bao et al., 2025] Wentao Bao, Lichang Chen, Heng Huang, and Yu Kong. Prompting language-informed distribution for compositional zero-shot learning. In European Confer- ence on Computer Vision, pages 107-123. Springer, 2025.","title":"Prompting language-informed distribution for compositional zero-shot learning","authors":"Wentao Bao, Lichang Chen, Heng Huang, Yu Kong","year":"2025","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Du et al., 2021] Jinlong Du, Senzhang Wang, Hao Miao, and Jiaqiang Zhang. Multi-channel pooling graph neural networks. In IJCAI, pages 1442-1448, 2021.","title":"Multi-channel pooling graph neural networks","authors":"Jinlong Du, Senzhang Wang, Hao Miao, Jiaqiang Zhang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/199"},{"position":3,"raw":"[Hao et al., 2023] Shaozhe Hao, Kai Han, and Kwan-Yee K Wong. Learning attention as disentangler for composi- tional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15315-15324, 2023.","title":"Learning attention as disentangler for compositional zero-shot learning","authors":"Hao","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2024] Siteng Huang, Biao Gong, Yutong Feng, Min Zhang, Yiliang Lv, and Donglin Wang. Troika: Multi-path cross-modal traction for compositional zero- shot learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 24005-24014, 2024.","title":"Troika: Multi-path cross-modal traction for compositional zeroshot learning","authors":"Siteng Huang, Biao Gong, Yutong Feng, Min Zhang, Yiliang Lv, Donglin Wang","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hudson and Manning, 2019] Drew A Hudson and Christo- pher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.","title":"Gqa: A new dataset for real-world visual reasoning and compositional question answering","authors":"A Drew, Christopher D Hudson, Manning","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Isola et al., 2015] Phillip Isola, Joseph J Lim, and Edward H Adelson. Discovering states and transformations in im- age collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1383- 1391, 2015.","title":"Discovering states and transformations in image collections","authors":"Phillip Isola, Joseph J Lim, Edward H Adelson","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jing et al., 2024] Chenchen Jing, Yukun Li, Hao Chen, and Chunhua Shen. Retrieval-augmented primitive represen- tations for compositional zero-shot learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 2652-2660, 2024.","title":"Retrieval-augmented primitive representations for compositional zero-shot learning","authors":"Chenchen Jing, Yukun Li, Hao Chen, Chunhua Shen","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karthik et al., 2022] Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Kg-sp: Knowledge guided simple primitives for open world compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9336- 9345, 2022.","title":"Kg-sp: Knowledge guided simple primitives for open world compositional zero-shot learning","authors":"Karthik","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Khan et al., 2023] Muhammad Gul Zain Ali Khan, Muham- mad Ferjad Naeem, Luc Van Gool, Alain Pagani, Didier Stricker, and Muhammad Zeshan Afzal. Learning atten- tion propagation for compositional zero-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 3828-3837, 2023.","title":"Learning attention propagation for compositional zero-shot learning","authors":"Khan","year":"2023","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kruse et al., 2022] Rudolf Kruse, Sanaz Mostaghim, Chris- tian Borgelt, Christian Braune, and Matthias Steinbrecher. Multi-layer perceptrons. In Computational intelligence: a methodological introduction, pages 53-124. Springer, 2022.","title":"Multi-layer perceptrons","authors":"Rudolf Kruse, Sanaz Mostaghim, Christian Borgelt, Christian Braune, Matthias Steinbrecher","year":"2022","venue":"Computational intelligence: a methodological introduction","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang. Siamese contrastive embedding net- work for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9326-9335, 2022.","title":"Siamese contrastive embedding network for compositional zero-shot learning","authors":"Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, Muli Yang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2023] Yun Li, Zhe Liu, Saurav Jha, and Lina Yao. Distilled reverse attention network for open-world compositional zero-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1782-1791, 2023.","title":"Distilled reverse attention network for open-world compositional zero-shot learning","authors":"Yun Li, Zhe Liu, Saurav Jha, Lina Yao","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2024a] Yicong Li, Xiangguo Sun, Hongxu Chen, Sixiao Zhang, Yu Yang, and Guandong Xu. Attention is not the only choice: Counterfactual reasoning for path- based explainable recommendation. IEEE Transactions on Knowledge and Data Engineering, 2024.","title":"Attention is not the only choice: Counterfactual reasoning for pathbased explainable recommendation","authors":"Li","year":"2024","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2024b] Yicong Li, Yu Yang, Jiannong Cao, Shuaiqi Liu, Haoran Tang, and Guandong Xu. Toward structure fairness in dynamic graph embedding: A trend- aware dual debiasing approach. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1701-1712, 2024.","title":"Toward structure fairness in dynamic graph embedding: A trendaware dual debiasing approach","authors":"Li","year":"2024","venue":"Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2024c] Yun Li, Zhe Liu, Hang Chen, and Lina Yao. Context-based and diversity-driven specificity in composi- tional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17037-17046, 2024.","title":"Context-based and diversity-driven specificity in compositional zero-shot learning","authors":"Li","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2024] Ruitong Liu, Yanbin Wang, Zhenhao Guo, Haitao Xu, Zhan Qin, Wenrui Ma, and Fan Zhang. Transurl: Improving malicious url detection with multi- layer transformer encoding and multi-scale pyramid fea- tures. Computer Networks, 253:110707, 2024.","title":"Transurl: Improving malicious url detection with multilayer transformer encoding and multi-scale pyramid features","authors":"Ruitong Liu, Yanbin Wang, Zhenhao Guo, Haitao Xu, Zhan Qin, Wenrui Ma, Fan Zhang","year":"2024","venue":"Computer Networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2023] Xiaocheng Lu, Song Guo, Ziming Liu, and Jingcai Guo. Decomposed soft prompt guided fusion en- hancing for compositional zero-shot learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23560-23569, 2023.","title":"Decomposed soft prompt guided fusion enhancing for compositional zero-shot learning","authors":"Lu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miao et al., 2025] Hao Miao, Ronghui Xu, Yan Zhao, Sen- zhang Wang, Jianxin Wang, Philip S Yu, and Christian S Jensen. A parameter-efficient federated framework for streaming time series anomaly detection via lightweight adaptation. TMC, 2025.","title":"A parameter-efficient federated framework for streaming time series anomaly detection via lightweight adaptation","authors":"Hao Miao, Ronghui Xu, Yan Zhao, Senzhang Wang, Jianxin Wang, Philip S Yu, Christian S Jensen","year":"2025","venue":"TMC","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Misra et al., 2017] Ishan Misra, Abhinav Gupta, and Mar- tial Hebert. From red wine to red tomato: Composition with context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1792- 1801, 2017.","title":"From red wine to red tomato: Composition with context","authors":"Ishan Misra, Abhinav Gupta, Martial Hebert","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Naeem et al., 2021] Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 953-962, 2021.","title":"Learning graph embeddings for compositional zero-shot learning","authors":"Naeem","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nagarajan and Grauman, 2018] Tushar Nagarajan and Kris- ten Grauman. Attributes as operators: factorizing unseen attribute-object compositions. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 169- 185, 2018.","title":"Attributes as operators: factorizing unseen attribute-object compositions","authors":"Tushar Nagarajan, Kristen Grauman","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nan et al., 2019] Zhixiong Nan, Yang Liu, Nanning Zheng, and Song-Chun Zhu. Recognizing unseen attribute-object pair with generative model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8811-8818, 2019.","title":"Recognizing unseen attribute-object pair with generative model","authors":"Nan","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nayak et al., 2022] Nihal V Nayak, Peilin Yu, and Stephen H Bach. Learning to compose soft prompts for compositional zero-shot learning. arXiv preprint arXiv:2204.03574, 2022.","title":"Learning to compose soft prompts for compositional zero-shot learning","authors":"Peilin Nihal V Nayak, Stephen H Yu, Bach","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Purushwalkam et al., 2019] Senthil Purushwalkam, Maxi- milian Nickel, Abhinav Gupta, and Marc'Aurelio Ranzato. Task-driven modular networks for zero-shot compositional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3593-3602, 2019.","title":"Task-driven modular networks for zero-shot compositional learning","authors":"Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, Marc'aurelio Ranzato","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Saini et al., 2022] Nirat Saini, Khoi Pham, and Abhinav Shrivastava. Disentangling visual embeddings for at- tributes and objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13658-13667, 2022.","title":"Disentangling visual embeddings for attributes and objects","authors":"Nirat Saini, Khoi Pham, Abhinav Shrivastava","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Scarselli et al., 2008] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.","title":"The graph neural network model","authors":"Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, Gabriele Monfardini","year":"2008","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tang et al., 2023] Feilong Tang, Zhongxing Xu, Qiming Huang, Jinfeng Wang, Xianxu Hou, Jionglong Su, and Jingxin Liu. Duat: Dual-aggregation transformer network for medical image segmentation. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 343-356. Springer, 2023.","title":"Duat: Dual-aggregation transformer network for medical image segmentation","authors":"Feilong Tang, Zhongxing Xu, Qiming Huang, Jinfeng Wang, Xianxu Hou, Jionglong Su, Jingxin Liu","year":"2023","venue":"Chinese Conference on Pattern Recognition and Computer Vision (PRCV)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022] Ruotong Wang, Yanqing Shen, Weiliang Zuo, Sanping Zhou, and Nanning Zheng. Transvpr: Transformer-based place recognition with multi-level at- tention aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13648-13657, 2022.","title":"Transvpr: Transformer-based place recognition with multi-level attention aggregation","authors":"Wang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2023a] Henan Wang, Muli Yang, Kun Wei, and Cheng Deng. Hierarchical prompt learning for composi- tional zero-shot recognition. In IJCAI, volume 1, page 3, 2023.","title":"Hierarchical prompt learning for compositional zero-shot recognition","authors":"Wang","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/163"},{"position":32,"raw":"[Wang et al., 2023b] Qingsheng Wang, Lingqiao Liu, Chenchen Jing, Hao Chen, Guoqiang Liang, Peng Wang, and Chunhua Shen. Learning conditional attributes for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11197-11206, 2023.","title":"Learning conditional attributes for compositional zero-shot learning","authors":"Wang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2024a] Guangyue Xu, Joyce Chai, and Parisa Ko- rdjamshidi. Gipcol: Graph-injected soft prompting for compositional zero-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 5774-5783, 2024.","title":"Gipcol: Graph-injected soft prompting for compositional zero-shot learning","authors":"Xu","year":"2024","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2024b] Ronghui Xu, Hao Miao, Senzhang Wang, Philip S Yu, and Jianxin Wang. Pefad: a parameter- efficient federated framework for time series anomaly de- tection. In SIGKDD, pages 3621-3632, 2024.","title":"Pefad: a parameterefficient federated framework for time series anomaly detection","authors":"Xu","year":"2024","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2023] Qu Yang, Mang Ye, Zhaohui Cai, Kehua Su, and Bo Du. Composed image retrieval via cross re- lation network with hierarchical aggregation transformer. IEEE Transactions on Image Processing, 32, 2023.","title":"Composed image retrieval via cross relation network with hierarchical aggregation transformer","authors":"Qu Yang, Mang Ye, Zhaohui Cai, Kehua Su, Bo Du","year":"2023","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yu and Grauman, 2014] Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 192-199, 2014.","title":"Fine-grained visual comparisons with local learning","authors":"Aron Yu, Kristen Grauman","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, page 2337-2348, Sep 2022.","title":"Learning to prompt for vision-language models","authors":"Zhou","year":"2022","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}