{"type":"ScholarlyArticle","title":"CATrans: Context and Affinity Transformer for Few-Shot Segmentation","authors":[{"name":"Shan Zhang","slug":"shan-zhang","given":"Shan","family":"Zhang","affiliation":"Australian National University , Canberra , Australia"},{"name":"Tianyi Wu","slug":"tianyi-wu","given":"Tianyi","family":"Wu","affiliation":"Institute of Deep Learning , Baidu Research , Beijing , China; National Engineering Laboratory for Deep Learning Technology and Application , Beijing , China"},{"name":"Sitong Wu","slug":"sitong-wu","given":"Sitong","family":"Wu","affiliation":"Institute of Deep Learning , Baidu Research , Beijing , China; National Engineering Laboratory for Deep Learning Technology and Application , Beijing , China"},{"name":"Guodong Guo","slug":"guodong-guo","given":"Guodong","family":"Guo","affiliation":"Institute of Deep Learning , Baidu Research , Beijing , China; National Engineering Laboratory for Deep Learning Technology and Application , Beijing , China"}],"abstract":"Few-shot segmentation (FSS) aims to segment novel categories given scarce annotated support images. The crux of FSS is how to aggregate dense correlations between support and query images for query segmentation while being robust to the large variations in appearance and context. To this end, previous Transformer-based methods explore global consensus either on context similarity or affinity map between support-query pairs. In this work, we effectively integrate the context and affinity information via the proposed novel Context and Affinity Transformer (CATrans) in a hierarchical architecture. Specifically, the Relation-guided Context Transformer (RCT) propagates context information from support to query images conditioned on more informative support features. Based on the observation that a huge feature distinction between support and query pairs brings barriers for context knowledge transfer, the Relation-guided Affinity Transformer (RAT) measures attention-aware affinity as auxiliary information for FSS, in which the self-affinity is responsible for more reliable cross-affinity. We conduct experiments to demonstrate the effectiveness of the proposed model, outperforming the state-of-the-art methods.","doi":"10.24963/ijcai.2022/231","url":"https://ijcai.org/proceedings/2022/231","pdf":"https://ijcai.org/proceedings/2022/0231.pdf","markdown":"https://ijcai.org/proceedings/2022/231.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/231","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":231,"pages":{"first":1658,"last":1664},"keywords":["Computer Vision: Machine Learning for Vision","Computer Vision: Transfer, low-shot, semi- and un- supervised learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986479","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984622","length_sec":346,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV 2020, volume 12346 of Lecture Notes in Computer Science, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Carion","year":"2020","venue":"ECCV 2020","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2021] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Semantic correspondence with transformers. CoRR, abs/2106.02520, 2021.","title":"Semantic correspondence with transformers","authors":"Cho","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong and Xing, 2018] Nanqing Dong and Eric P. Xing. Few-shot semantic segmentation with prototype learning. In BMVC 2018, page 79. BMVA Press, 2018.","title":"Few-shot semantic segmentation with prototype learning","authors":"Dong, Xing ;, Nanqing Dong, Eric P Xing","year":"2018","venue":"BMVC 2018","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR 2021. OpenReview.net, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR 2021","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hariharan et al., 2011] Bharath Hariharan, Pablo Arbelaez, Lubomir D. Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV 2011, pages 991-998. IEEE Computer Society, 2011.","title":"Semantic contours from inverse detectors","authors":"Bharath Hariharan, Pablo Arbelaez, D Lubomir, Subhransu Bourdev, Jitendra Maji, Malik","year":"2011","venue":"ICCV 2011","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Johnander et al., 2021] Joakim Johnander, Johan Edstedt, Michael Felsberg, Fahad Shahbaz Khan, and Martin Danelljan. Dense gaussian processes for few-shot segmen- tation. CoRR, abs/2110.03674, 2021.","title":"Dense gaussian processes for few-shot segmentation","authors":"Johnander","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: com- mon objects in context. In ECCV 2014, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"ECCV 2014","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin et al., 2017] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian D. Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR 2017, pages 5168-5177. IEEE Computer Society, 2017.","title":"Refinenet: Multi-path refinement networks for high-resolution semantic segmentation","authors":"Guosheng Lin, Anton Milan, Chunhua Shen, Ian D Reid","year":"2017","venue":"CVPR 2017","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2020] Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In ECCV 2020, volume 12354 of Lecture Notes in Computer Science, pages 142- 158. Springer, 2020.","title":"Part-aware prototype network for few-shot semantic segmentation","authors":"Liu","year":"2020","venue":"ECCV 2020","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR 2015, pages 3431-3440. IEEE Computer Society, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR 2015","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu et al., 2021] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. CoRR, abs/2108.03032, 2021.","title":"Simpler is better: Few-shot semantic segmentation with classifier weight transformer","authors":"Lu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nguyen and Todorovic, 2019] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In ICCV 2019, pages 622-631. IEEE, 2019.","title":"Feature weighting and boosting for few-shot segmentation","authors":"Todorovic Nguyen, Khoi Nguyen, Sinisa Todorovic","year":"2019","venue":"ICCV 2019","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shaban et al., 2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In BMVC 2017. BMVA Press, 2017.","title":"One-shot learning for semantic segmentation","authors":"Shaban","year":"2017","venue":"BMVC 2017","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tian et al., 2020] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. CoRR, abs/2008.01449, 2020.","title":"Prior guided feature enrichment network for few-shot segmentation","authors":"Tian","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: December 4-9, 2017, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2019] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot im- age semantic segmentation with prototype alignment. In ICCV 2019, pages 9196-9205. IEEE, 2019.","title":"Panet: Few-shot image semantic segmentation with prototype alignment","authors":"Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng","year":"2019","venue":"ICCV 2019","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2020] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen. Few- shot semantic segmentation with democratic attention net- works. In ECCV 2020, volume 12358 of Lecture Notes in Computer Science, pages 730-746. Springer, 2020.","title":"Fewshot semantic segmentation with democratic attention networks","authors":"Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, Xiantong Zhen","year":"2020","venue":"ECCV 2020","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2020] Boyu Yang, Chang Liu, Bohao Li, Jian- bin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In ECCV 2020, volume 12353 of Lecture Notes in Computer Science, pages 763- 778. Springer, 2020.","title":"Prototype mixture models for few-shot semantic segmentation","authors":"Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, Qixiang Ye","year":"2020","venue":"ECCV 2020","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang et al., 2019a] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region-based one- shot semantic segmentation. In ICCV 2019, pages 9586- 9594. IEEE, 2019.","title":"Pyramid graph networks with connection attentions for region-based oneshot semantic segmentation","authors":"Zhang","year":"2019","venue":"ICCV 2019","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2019b] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic seg- mentation networks with iterative refinement and atten- tive few-shot learning. In CVPR 2019, pages 5217-5226. Computer Vision Foundation / IEEE, 2019.","title":"Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning","authors":"Zhang","year":"2019","venue":"CVPR 2019","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2020] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S. Huang. Sg-one: Similarity guidance net- work for one-shot semantic segmentation. IEEE Trans. Cybern., 50(9):3855-3865, 2020.","title":"Sg-one: Similarity guidance network for one-shot semantic segmentation","authors":"Xiaolin Zhang, Yunchao Wei, Yi Yang, Thomas S Huang","year":"2020","venue":"IEEE Trans. Cybern","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2021] Gengwei Zhang, Guoliang Kang, Yun- chao Wei, and Yi Yang. Few-shot segmentation via cycle- consistent transformer. CoRR, abs/2106.02320, 2021.","title":"Few-shot segmentation via cycleconsistent transformer","authors":"Gengwei Zhang, Guoliang Kang, Yunchao Wei, Yi Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2021] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: de- formable transformers for end-to-end object detection. In ICLR 2021. OpenReview.net, 2021.","title":"Deformable DETR: deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2021","venue":"OpenReview.net","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}