{"type":"ScholarlyArticle","title":"A Transformer-Based Adaptive Prototype Matching Network for Few-Shot Semantic Segmentation","authors":[{"name":"Sihan Chen","slug":"sihan-chen","given":"Sihan","family":"Chen","affiliation":"School of Computer Science , Nanjing University of Information Science and Technology"},{"name":"Yadang Chen","slug":"yadang-chen","given":"Yadang","family":"Chen","affiliation":"School of Computer Science , Nanjing University of Information Science and Technology"},{"name":"Yuhui Zheng","slug":"yuhui-zheng","given":"Yuhui","family":"Zheng","affiliation":"College of Computer , Qinghai Normal University"},{"name":"Zhi-Xin Yang","slug":"zhi-xin-yang","given":"Zhi-Xin","family":"Yang","affiliation":"State Key Laboratory of Internet of Things for Smart City , University of Macau"},{"name":"Enhua Wu","slug":"enhua-wu","given":"Enhua","family":"Wu","affiliation":"Key Laboratory of System Software and State Key Laboratory of Computer Science , Institute of Software , Chinese Academy of Sciences"}],"abstract":"Few-shot semantic segmentation (FSS) aims to generate a model for segmenting novel classes using a limited number of annotated samples. Previous FSS methods have shown sensitivity to background noise due to inherent bias, attention bias, and spatial-aware bias. In this study, we propose a Transformer-Based Adaptive Prototype Matching Network to establish robust matching relationships by improving the semantic and spatial perception of query features. The model includes three modules: target enhancement module (TEM), dual constraint aggregation module (DCAM), and dual classification module (DCM). In particular, TEM mitigates inherent bias by exploring the relevance of multi-scale local context to enhance foreground features. Then, DCAM addresses attention bias through the dual semantic-aware attention mechanism to strengthen constraints. Finally, the DCM module decouples the segmentation task into semantic alignment and spatial alignment to alleviate spatial-aware bias. Extensive experiments on PASCAL-5i and COCO-20i confirm the effectiveness of our approach.","doi":"10.24963/ijcai.2024/73","url":"https://ijcai.org/proceedings/2024/73","pdf":"https://ijcai.org/proceedings/2024/0073.pdf","markdown":"https://ijcai.org/proceedings/2024/73.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/73","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":73,"pages":{"first":659,"last":667},"keywords":["Computer Vision: CV: Scene analysis and understanding","Computer Vision: CV: Segmentation","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Bao et al., 2023] Xiaoyi Bao, Jie Qin, Siyang Sun, Yun Zheng, and Xingang Wang. Relevant intrinsic feature en- hancement network for few-shot semantic segmentation. arXiv preprint arXiv:2312.06474, 2023.","title":"Relevant intrinsic feature enhancement network for few-shot semantic segmentation","authors":"Xiaoyi Bao, Jie Qin, Siyang Sun, Yun Zheng, Xingang Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2022] Leilei Cao, Yibo Guo, Ye Yuan, and Qiangguo Jin. Prototype as query for few shot semantic segmentation. arXiv preprint arXiv:2211.14764, 2022.","title":"Prototype as query for few shot semantic segmentation","authors":"Leilei Cao, Yibo Guo, Ye Yuan, Qiangguo Jin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2022a] Jiacheng Chen, Bin-Bin Gao, Zongqing Lu, Jing-Hao Xue, Chengjie Wang, and Qingmin Liao. Apanet: Adaptive prototypes alignment network for few- shot semantic segmentation. IEEE Transactions on Multi- media, pages 1-1, 2022.","title":"Apanet: Adaptive prototypes alignment network for fewshot semantic segmentation","authors":"Chen","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022b] Yadang Chen, Chuanyan Hao, Zhi-Xin Yang, and Enhua Wu. Fast target-aware learning for few- shot video object segmentation. Science China Informa- tion Sciences, 65(8):182104, 2022.","title":"Fast target-aware learning for fewshot video object segmentation","authors":"Chen","year":"2022","venue":"Science China Information Sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2024] Yadang Chen, Ren Jiang, Yuhui Zheng, Bin Sheng, Zhi-Xin Yang, and Enhua Wu. Dual branch multi-level semantic learning for few-shot segmentation. IEEE Transactions on Image Processing, 33:1432-1447, 2024.","title":"Dual branch multi-level semantic learning for few-shot segmentation","authors":"Yadang Chen, Ren Jiang, Yuhui Zheng, Bin Sheng, Zhi-Xin Yang, Enhua Wu","year":"2024","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheng et al., 2022] Bowen Cheng, Ishan Misra, Alexan- der G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 1290-1299, 2022.","title":"Masked-attention mask transformer for universal image segmentation","authors":"Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, Rohit Girdhar","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.","title":"Density estimation using real nvp","authors":"Dinh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong and Xing, 2018] Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning. In BMVC, volume 3, 2018.","title":"Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning","authors":"Xing Dong","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fan et al., 2022] Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi- Keung Tang. Self-support few-shot semantic segmenta- tion. arXiv preprint arXiv:2207.11549, 2022.","title":"Self-support few-shot semantic segmentation","authors":"Qi Fan, Wenjie Pei, Yu-Wing Tai, Chi-Keung Tang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2022] Wei-Jie Huang, Yu-Lin Lu, Shih-Yao Lin, Yusheng Xie, and Yen-Yu Lin. Aqt: Adversarial query transformers for domain adaptive object detection. In 31st International Joint Conference on Artificial Intel- ligence, IJCAI 2022, pages 972-979. International Joint Conferences on Artificial Intelligence, 2022.","title":"Aqt: Adversarial query transformers for domain adaptive object detection","authors":"Wei-Jie Huang, Yu-Lin Lu, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin","year":"2022","venue":"31st International Joint Conference on Artificial Intelligence, IJCAI 2022","doi":null,"url":null,"internal_url":"/proceedings/2022/136"},{"position":11,"raw":"[Lang et al., 2022] Chunbo Lang, Gong Cheng, Binfei Tu, and Junwei Han. Learning what not to segment: A new perspective on few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 8057-8067, 2022.","title":"Learning what not to segment: A new perspective on few-shot segmentation","authors":"Lang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adap- tive prototype learning and allocation for few-shot seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8334- 8343, 2021.","title":"Adaptive prototype learning and allocation for few-shot segmentation","authors":"Li","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2023] Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 6015-6026, 2023.","title":"Scale-aware modulation meet transformer","authors":"Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, Lianwen Jin","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2022a] Jie Liu, Yanqi Bao, Guo-Sen Xie, Huan Xiong, Jan-Jakob Sonke, and Efstratios Gavves. Dy- namic prototype convolution network for few-shot seman- tic segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 11553-11562, 2022.","title":"Dynamic prototype convolution network for few-shot semantic segmentation","authors":"Liu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2022b] Yuanwei Liu, Nian Liu, Qinglong Cao, Xiwen Yao, Junwei Han, and Ling Shao. Learning non-target knowledge for few-shot semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11573- 11582, 2022.","title":"Learning non-target knowledge for few-shot semantic segmentation","authors":"Liu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2022c] Yuanwei Liu, Nian Liu, Xiwen Yao, and Junwei Han. Intermediate prototype mining transformer for few-shot semantic segmentation. Advances in Neural Information Processing Systems, 35:38020-38031, 2022.","title":"Intermediate prototype mining transformer for few-shot semantic segmentation","authors":"Liu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2021] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8741-8750, 2021.","title":"Simpler is better: Few-shot semantic segmentation with classifier weight transformer","authors":"Lu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Min et al., 2021] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6941-6952, 2021.","title":"Hypercorrelation squeeze for few-shot segmentation","authors":"Min","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nguyen and Todorovic, 2019] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 622-631, 2019.","title":"Feature weighting and boosting for few-shot segmentation","authors":"Todorovic Nguyen, Khoi Nguyen, Sinisa Todorovic","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ouyang et al., 2023] Shuyi Ouyang, Hongyi Wang, Shiao Xie, Ziwei Niu, Ruofeng Tong, Yen-Wei Chen, and Lanfen Lin. Slvit: scale-wise language-guided vision transformer for referring image segmentation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 1294-1302, 2023.","title":"Slvit: scale-wise language-guided vision transformer for referring image segmentation","authors":"Shuyi Ouyang, Hongyi Wang, Shiao Xie, Ziwei Niu, Ruofeng Tong, Yen-Wei Chen, Lanfen Lin","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23","doi":null,"url":null,"internal_url":"/proceedings/2023/144"},{"position":21,"raw":"[Peng et al., 2023] Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 23641-23651, 2023.","title":"Hierarchical dense correlation distillation for few-shot segmentation","authors":"Peng","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shaban et al., 2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410, 2017.","title":"One-shot learning for semantic segmentation","authors":"Shaban","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shi et al., 2022] Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. Dense cross-query-and-support attention weighted mask aggregation for few-shot seg- mentation. In European Conference on Computer Vision, pages 151-168. Springer, 2022.","title":"Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation","authors":"Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, Yefeng Zheng","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tian et al., 2020] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 2020.","title":"Prior guided feature enrichment network for few-shot segmentation","authors":"Tian","year":"2020","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2019] Bin Wang, Guojun Qi, Sheng Tang, Tianzhu Zhang, Yunchao Wei, Linghui Li, and Yong- dong Zhang. Boundary perception guidance: A scribble- supervised semantic segmentation approach. In IJCAI International joint conference on artificial intelligence, 2019.","title":"Boundary perception guidance: A scribblesupervised semantic segmentation approach","authors":"Wang","year":"2019","venue":"IJCAI International joint conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Woo et al., 2018] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European confer- ence on computer vision (ECCV), pages 3-19, 2018.","title":"Cbam: Convolutional block attention module","authors":"Sanghyun Woo, Jongchan Park, Joon-Young Lee, In So Kweon","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2021] Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao. Scale-aware graph neural network for few-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5475-5484, 2021.","title":"Scale-aware graph neural network for few-shot semantic segmentation","authors":"Guo-Sen Xie, Jie Liu, Huan Xiong, Ling Shao","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2020] Boyu Yang, Chang Liu, Bohao Li, Jian- bin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In European Conference on Computer Vision, pages 763-778. Springer, 2020.","title":"Prototype mixture models for few-shot semantic segmentation","authors":"Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, Qixiang Ye","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2021] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. Mining latent classes for few-shot seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8721-8730, 2021.","title":"Mining latent classes for few-shot segmentation","authors":"Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, Yang Gao","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2023] Yong Yang, Qiong Chen, Yuan Feng, and Tianlin Huang. Mianet: Aggregating unbiased instance and general information for few-shot semantic segmen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7131- 7140, 2023.","title":"Mianet: Aggregating unbiased instance and general information for few-shot semantic segmentation","authors":"Yong Yang, Qiong Chen, Yuan Feng, Tianlin Huang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ye et al., 2021] Qiaolin Ye, Peng Huang, Zhao Zhang, Yuhui Zheng, Liyong Fu, and Wankou Yang. Multiview learning with robust double-sided twin svm. IEEE trans- actions on Cybernetics, 52(12):12745-12758, 2021.","title":"Multiview learning with robust double-sided twin svm","authors":"Ye","year":"2021","venue":"IEEE transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2019] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region- based one-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 9587-9595, 2019.","title":"Pyramid graph networks with connection attentions for regionbased one-shot semantic segmentation","authors":"Zhang","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2020] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance net- work for one-shot semantic segmentation. IEEE transac- tions on cybernetics, 50(9):3855-3865, 2020.","title":"Sg-one: Similarity guidance network for one-shot semantic segmentation","authors":"Xiaolin Zhang, Yunchao Wei, Yi Yang, Thomas S Huang","year":"2020","venue":"IEEE transactions on cybernetics","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2021a] Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross-guided learning for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8312-8321, 2021.","title":"Self-guided and cross-guided learning for few-shot segmentation","authors":"Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2021b] Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Infor- mation Processing Systems, 34:21984-21996, 2021.","title":"Few-shot segmentation via cycle-consistent transformer","authors":"Zhang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}