{"type":"ScholarlyArticle","title":"Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation","authors":[{"name":"Chunbo Lang","slug":"chunbo-lang","given":"Chunbo","family":"Lang","affiliation":"School of Automation , Northwestern Polytechnical University , Xi'an , China"},{"name":"Binfei Tu","slug":"binfei-tu","given":"Binfei","family":"Tu","affiliation":"School of Automation , Northwestern Polytechnical University , Xi'an , China"},{"name":"Gong Cheng","slug":"gong-cheng","given":"Gong","family":"Cheng","affiliation":""},{"name":"Junwei Han","slug":"junwei-han","given":"Junwei","family":"Han","affiliation":"School of Automation , Northwestern Polytechnical University , Xi'an , China"}],"abstract":"Few-shot segmentation, which aims to segment unseen-class objects given only a handful of densely labeled samples, has received widespread attention from the community. Existing approaches typically follow the prototype learning paradigm to perform meta-inference, which fails to fully exploit the underlying information from support image-mask pairs, resulting in various segmentation failures, e.g., incomplete objects, ambiguous boundaries, and distractor activation. To this end, we propose a simple yet versatile framework in the spirit of divide-and-conquer. Specifically, a novel self-reasoning scheme is first implemented on the annotated support image, and then the coarse segmentation mask is divided into multiple regions with different properties. Leveraging effective masked average pooling operations, a series of support-induced proxies are thus derived, each playing a specific role in conquering the above challenges. Moreover, we devise a unique parallel decoder structure that integrates proxies with similar attributes to boost the discrimination power. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information as a guide at the “episode” level, not just about the object cues themselves. Extensive experiments on PASCAL-5i and COCO-20i demonstrate the superiority of DCP over conventional prototype-based approaches (up to 5~10% on average), which also establishes a new state-of-the-art. Code is available at github.com/chunbolang/DCP.","doi":"10.24963/ijcai.2022/143","url":"https://ijcai.org/proceedings/2022/143","pdf":"https://ijcai.org/proceedings/2022/0143.pdf","markdown":"https://ijcai.org/proceedings/2022/143.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/143","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":143,"pages":{"first":1024,"last":1030},"keywords":["Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Segmentation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986971","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985114","length_sec":484,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Ia- sonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Se- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834-848, 2017.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2017","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019] Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert. Image deformation meta- networks for one-shot learning. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 8680-8689, 2019.","title":"Image deformation metanetworks for one-shot learning","authors":"Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, Martial Hebert","year":"2019","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis., 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep net- works. In Proc. IEEE Int. Conf. Mach. Learn., pages 1126-1135, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"Proc. IEEE Int. Conf. Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hariharan et al., 2011] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Seman- tic contours from inverse detectors. In Proc. IEEE Int. Conf. Comput. Vis., pages 991-998, 2011.","title":"Semantic contours from inverse detectors","authors":"Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, Jitendra Malik","year":"2011","venue":"Proc. IEEE Int. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2019] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In Proc. IEEE Int. Conf. Comput. Vis., pages 603-612, 2019.","title":"Ccnet: Criss-cross attention for semantic segmentation","authors":"Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, Wenyu Liu","year":"2019","venue":"Proc. IEEE Int. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lang et al., 2022] Chunbo Lang, Gong Cheng, Binfei Tu, and Jun- wei Han. Learning what not to segment: A new perspective on few-shot segmentation. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2022.","title":"Learning what not to segment: A new perspective on few-shot segmentation","authors":"Lang","year":"2022","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Gen Li, Varun Jampani, Laura Sevilla-Lara, De- qing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive proto- type learning and allocation for few-shot segmentation. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 8334- 8343, 2021.","title":"Adaptive prototype learning and allocation for few-shot segmentation","authors":"Li","year":"2021","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in con- text. In Proc. Eur. Conf. Comput. Vis., pages 740-755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Proc. Eur. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2021] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Simpler is better: Few-shot semantic seg- mentation with classifier weight transformer. In Proc. IEEE Int. Conf. Comput. Vis., pages 8741-8750, 2021.","title":"Simpler is better: Few-shot semantic segmentation with classifier weight transformer","authors":"Lu","year":"2021","venue":"Proc. IEEE Int. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nguyen and Todorovic, 2019] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proc. IEEE Int. Conf. Comput. Vis., pages 622-631, 2019.","title":"Feature weighting and boosting for few-shot segmentation","authors":"Todorovic Nguyen, Khoi Nguyen, Sinisa Todorovic","year":"2019","venue":"Proc. IEEE Int. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proc. Conf. Adv. Neural Inform. Process. Syst., volume 28, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Proc. Conf. Adv. Neural Inform. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexan- der C. Berg, and Li Fei-Fei. Imagenet large scale visual recogni- tion challenge. Int. J. Comput. Vis., 115:211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shaban et al., 2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Ir- fan Essa, and Byron Boots. One-shot learning for semantic seg- mentation. In Proc. Br. Mach. Vis. Conf., 2017.","title":"One-shot learning for semantic segmentation","authors":"Shaban","year":"2017","venue":"Proc. Br. Mach. Vis. Conf","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Siam et al., 2019] Mennatullah Siam, Boris N Oreshkin, and Mar- tin Jagersand. Amp: Adaptive masked proxies for few-shot seg- mentation. In Proc. IEEE Int. Conf. Comput. Vis., pages 5249- 5258, 2019.","title":"Mennatullah Siam, Boris N Oreshkin, and Martin Jagersand. Amp: Adaptive masked proxies for few-shot segmentation","authors":"Siam","year":"2019","venue":"Proc. IEEE Int. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proc. Conf. Adv. Neural Inform. Process. Syst., volume 30, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"Proc. Conf. Adv. Neural Inform. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tian et al., 2022] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature en- richment network for few-shot segmentation. IEEE Trans. Pat- tern Anal. Mach. Intell., 44(2):1050-1065, 2022.","title":"Prior guided feature enrichment network for few-shot segmentation","authors":"Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, Jiaya Jia","year":"2022","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Timothy Lil- licrap, koray kavukcuoglu, and Daan Wierstra. Matching net- works for one shot learning. In Proc. Conf. Adv. Neural Inform. Process. Syst., volume 29, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra","year":"2016","venue":"Proc. Conf. Adv. Neural Inform. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proc. IEEE Int. Conf. Comput. Vis., pages 9197-9206, 2019.","title":"Panet: Few-shot image semantic segmentation with prototype alignment","authors":"Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng","year":"2019","venue":"Proc. IEEE Int. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2021] Zhonghua Wu, Xiangxi Shi, Guosheng Lin, and Jianfei Cai. Learning meta-class memory for few-shot semantic segmentation. In Proc. IEEE Int. Conf. Comput. Vis., pages 517- 526, 2021.","title":"Learning meta-class memory for few-shot semantic segmentation","authors":"Wu","year":"2021","venue":"Proc. IEEE Int. Conf. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2021] Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao. Scale-aware graph neural network for few-shot seman- tic segmentation. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 5475-5484, 2021.","title":"Scale-aware graph neural network for few-shot semantic segmentation","authors":"Guo-Sen Xie, Jie Liu, Huan Xiong, Ling Shao","year":"2021","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learn- ing. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 5217-5226, 2019.","title":"Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning","authors":"Zhang","year":"2019","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one- shot semantic segmentation. IEEE Trans. Cybern., 50(9):3855- 3865, 2020.","title":"Sg-one: Similarity guidance network for oneshot semantic segmentation","authors":"Xiaolin Zhang, Yunchao Wei, Yi Yang, Thomas S Huang","year":"2020","venue":"IEEE Trans. Cybern","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2021] Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross-guided learning for few-shot segmentation. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 8312-8321, 2021.","title":"Self-guided and cross-guided learning for few-shot segmentation","authors":"Zhang","year":"2021","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pages 2881-2890, 2017.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}