{"type":"ScholarlyArticle","title":"Iterative Few-shot Semantic Segmentation from Image Label Text","authors":[{"name":"Haohan Wang","slug":"haohan-wang","given":"Haohan","family":"Wang","affiliation":"Shenzhen International Graduate School , Tsinghua University"},{"name":"Liang Liu","slug":"liang-liu","given":"Liang","family":"Liu","affiliation":"Tencent Youtu Lab"},{"name":"Wuhao Zhang","slug":"wuhao-zhang","given":"Wuhao","family":"Zhang","affiliation":"Tencent Youtu Lab"},{"name":"Jiangning Zhang","slug":"jiangning-zhang","given":"Jiangning","family":"Zhang","affiliation":"Tencent Youtu Lab"},{"name":"Zhenye Gan","slug":"zhenye-gan","given":"Zhenye","family":"Gan","affiliation":"Tencent Youtu Lab"},{"name":"Yabiao Wang","slug":"yabiao-wang","given":"Yabiao","family":"Wang","affiliation":"Tencent Youtu Lab"},{"name":"Chengjie Wang","slug":"chengjie-wang","given":"Chengjie","family":"Wang","affiliation":"Tencent Youtu Lab"},{"name":"Haoqian Wang","slug":"haoqian-wang","given":"Haoqian","family":"Wang","affiliation":"Shenzhen International Graduate School , Tsinghua University"}],"abstract":"Few-shot semantic segmentation aims to learn to segment unseen class objects with the guidance of only a few support images. Most previous methods rely on the pixel-level label of support images. In this paper, we focus on a more challenging setting, in which only the image-level labels are available. We propose a general framework to firstly generate coarse masks with the help of the powerful vision-language model CLIP, and then iteratively and mutually refine the mask predictions of support and query images. Extensive experiments on PASCAL-5i and COCO-20i datasets demonstrate that our method not only outperforms the state-of-the-art weakly supervised approaches by a significant margin, but also achieves comparable or better results to recent supervised methods. Moreover, our method owns an excellent generalization ability for the images in the wild and uncommon classes. Code will be available at https://github.com/Whileherham/IMR-HSNet.","doi":"10.24963/ijcai.2022/193","url":"https://ijcai.org/proceedings/2022/193","pdf":"https://ijcai.org/proceedings/2022/0193.pdf","markdown":"https://ijcai.org/proceedings/2022/193.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/193","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":193,"pages":{"first":1385,"last":1392},"keywords":["Machine Learning: Few-shot learning","Computer Vision: Segmentation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986990","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985133","length_sec":654,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Boudiaf et al., 2021] Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few-shot segmentation without meta- learning: A good transductive inference is all you need? In CVPR, 2021.","title":"Few-shot segmentation without metalearning: A good transductive inference is all you need?","authors":"Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, Jose Dolz","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bucher et al., 2019] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. NeurIPS, 2019.","title":"Zero-shot semantic segmentation","authors":"Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, Patrick Pérez","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gao et al., 2021] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv:2110.04544, 2021.","title":"Clip-adapter: Better vision-language models with feature adapters","authors":"Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lee et al., 2022] Yuan-Hao Lee, Fu-En Yang, and Yu- Chiang Frank Wang. A pixel-level meta-learner for weakly supervised few-shot semantic segmentation. In WACV, 2022.","title":"A pixel-level meta-learner for weakly supervised few-shot semantic segmentation","authors":"Yuan-Hao Lee, Fu-En, Yu-Chiang Yang, Wang Frank","year":"2022","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Li et al., 2021a] Gen Li, Varun Jampani, Laura Sevilla- Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. In CVPR, 2021.","title":"Adaptive prototype learning and allocation for few-shot segmentation","authors":"Li","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2021b] Yiwen Li, Gratianus Wesley Putra Data, Yunguan Fu, Yipeng Hu, and Victor Adrian Prisacariu. Few-shot semantic segmentation with self-supervision from pseudo-classes. arXiv:2110.11742, 2021.","title":"Few-shot semantic segmentation with self-supervision from pseudo-classes","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2022] Boyi Li, Kilian Q Weinberger, Serge Be- longie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In ICLR, 2022.","title":"Language-driven semantic segmentation","authors":"Boyi Li, Q Kilian, Serge Weinberger, Vladlen Belongie, Rene Koltun, Ranftl","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2021] Binghao Liu, Yao Ding, Jianbin Jiao, Xi- angyang Ji, and Qixiang Ye. Anti-aliasing semantic recon- struction for few-shot semantic segmentation. In CVPR, 2021.","title":"Anti-aliasing semantic reconstruction for few-shot semantic segmentation","authors":"Binghao Liu, Yao Ding, Jianbin Jiao, Xiangyang Ji, Qixiang Ye","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu et al., 2021] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In ICCV, 2021.","title":"Simpler is better: Few-shot semantic segmentation with classifier weight transformer","authors":"Lu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Min et al., 2021] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In ICCV, 2021.","title":"Hypercorrelation squeeze for few-shot segmentation","authors":"Min","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rao et al., 2021] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense pre- diction with context-aware prompting. arXiv:2112.01518, 2021.","title":"Denseclip: Language-guided dense prediction with context-aware prompting","authors":"Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, Jiwen Lu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Raza et al., 2019] Hasnain Raza, Mahdyar Ravanbakhsh, Tassilo Klein, and Moin Nabi. Weakly supervised one shot segmentation. In ICCVW, 2019.","title":"Weakly supervised one shot segmentation","authors":"Hasnain Raza, Mahdyar Ravanbakhsh, Tassilo Klein, Moin Nabi","year":"2019","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shaban et al., 2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In BMVC, 2017.","title":"One-shot learning for semantic segmentation","authors":"Shaban","year":"2017","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Siam et al., 2020] Mennatullah Siam, Naren Doraiswamy, Boris N Oreshkin, Hengshuai Yao, and Martin Jagersand. Weakly supervised few-shot object segmentation with vi- sual and semantic embeddings. In IJCAI, 2020.","title":"Weakly supervised few-shot object segmentation with visual and semantic embeddings","authors":"Siam","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2021] Guolei Sun, Yun Liu, Jingyun Liang, and Luc Van Gool. Boosting few-shot semantic segmentation with transformers. arXiv:2108.02266, 2021.","title":"Boosting few-shot semantic segmentation with transformers","authors":"Guolei Sun, Yun Liu, Jingyun Liang, Luc Van Gool","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tian et al., 2020] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. TPAMI, 2020.","title":"Prior guided feature enrichment network for few-shot segmentation","authors":"Tian","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot im- age semantic segmentation with prototype alignment. In ICCV, 2019.","title":"Panet: Few-shot image semantic segmentation with prototype alignment","authors":"Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021] Zhonghua Wu, Xiangxi Shi, Guosheng Lin, and Jianfei Cai. Learning meta-class memory for few- shot semantic segmentation. In ICCV, 2021.","title":"Learning meta-class memory for fewshot semantic segmentation","authors":"Wu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2021] Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao. Scale-aware graph neural network for few-shot semantic segmentation. In CVPR, 2021.","title":"Scale-aware graph neural network for few-shot semantic segmentation","authors":"Guo-Sen Xie, Jie Liu, Huan Xiong, Ling Shao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2020] Boyu Yang, Chang Liu, Bohao Li, Jian- bin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In ECCV, 2020.","title":"Prototype mixture models for few-shot semantic segmentation","authors":"Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, Qixiang Ye","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2021] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. Mining latent classes for few-shot segmentation. In ICCV, 2021.","title":"Mining latent classes for few-shot segmentation","authors":"Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, Yang Gao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic seg- mentation networks with iterative refinement and attentive few-shot learning. In CVPR, 2019.","title":"Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning","authors":"Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2021] Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross-guided learning for few-shot segmentation. In CVPR, 2021.","title":"Self-guided and cross-guided learning for few-shot segmentation","authors":"Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2021] Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv:2112.01071, 2021.","title":"Denseclip: Extract free dense labels from clip","authors":"Chong Zhou, Chen Change Loy, Bo Dai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}