{"type":"ScholarlyArticle","title":"Self-Supervised Tuning for Few-Shot Segmentation","authors":[{"name":"Kai Zhu","slug":"kai-zhu","given":"Kai","family":"Zhu","affiliation":"University of Science and Technology of China"},{"name":"Wei Zhai","slug":"wei-zhai","given":"Wei","family":"Zhai","affiliation":"University of Science and Technology of China"},{"name":"Yang Cao","slug":"yang-cao","given":"Yang","family":"Cao","affiliation":"University of Science and Technology of China"}],"abstract":"Few-shot segmentation aims at assigning a category label to each image pixel with few annotated samples. It is a challenging task since the dense prediction can only be achieved under the guidance of latent features defined by sparse annotations. Existing meta-learning based method tends to fail in generating category-specifically discriminative descriptor when the visual features extracted from support images are marginalized in embedding space. To address this issue, this paper presents an adaptive tuning framework, in which the distribution of latent features across different episodes is dynamically adjusted based on a self-segmentation scheme, augmenting category-specific descriptors for label prediction. Specifically, a novel self-supervised inner-loop is firstly devised as the base learner to extract the underlying semantic features from the support image. Then, gradient maps are calculated by back-propagating self-supervised loss through the obtained features, and leveraged as guidance for augmenting the corresponding elements in the embedding space. Finally, with the ability to continuously learn from different episodes, an optimization-based meta-learner is adopted as outer loop of our proposed framework to gradually refine the segmentation results. Extensive experiments on benchmark PASCAL-5i and COCO-20i datasets demonstrate the superiority of our proposed method over state-of-the-art.","doi":"10.24963/ijcai.2020/142","url":"https://ijcai.org/proceedings/2020/142","pdf":"https://ijcai.org/proceedings/2020/0142.pdf","markdown":"https://ijcai.org/proceedings/2020/142.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/142","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":142,"pages":{"first":1019,"last":1025},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Other","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24436/file_24436.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Bertinetto et al., 2018] Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.08136, 2018.","title":"Meta-learning with differentiable closed-form solvers","authors":"Luca Bertinetto, Joao F Henriques, Philip Hs Torr, Andrea Vedaldi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.","title":"Rethinking atrous convolution for semantic image segmentation","authors":"Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Zhe Chen, Jing Zhang, and Dacheng Tao. Progressive lidar adaptation for road detection. IEEE/CAA Journal of Automatica Sinica, 6(3):693-702, 2019.","title":"Progressive lidar adaptation for road detection","authors":"Zhe Chen, Jing Zhang, Dacheng Tao","year":"2019","venue":"IEEE/CAA Journal of Automatica Sinica","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong and Xing, 2018] Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning. In BMVC, volume 1, page 6, 2018.","title":"Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning","authors":"Xing Dong","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2019] Tao Hu, Pengwan Yang, Chiliang Zhang, Gang Yu, Yadong Mu, and Cees GM Snoek. Attention- based multi-context guiding for few-shot semantic seg- mentation. In AAAI, volume 33, pages 8441-8448, 2019.","title":"Attentionbased multi-context guiding for few-shot semantic segmentation","authors":"Tao Hu, Pengwan Yang, Chiliang Zhang, Gang Yu, Yadong Mu, G M Cees, Snoek","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Koch et al., 2015] Gregory Koch, Richard Zemel, and Rus- lan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, vol- ume 2. Lille, 2015.","title":"Siamese neural networks for one-shot image recognition","authors":"Gregory Koch, Richard Zemel, Ruslan Salakhutdinov","year":"2015","venue":"ICML deep learning workshop","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lee et al., 2019] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with dif- ferentiable convex optimization. In CVPR, pages 10657- 10665, 2019.","title":"Meta-learning with differentiable convex optimization","authors":"Lee","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Munkhdalai and Yu, 2017] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2554-2563. JMLR. org, 2017.","title":"Tsendsuren Munkhdalai and Hong Yu. Meta networks","authors":"Yu Munkhdalai","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nguyen and Todorovic, 2019] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In ICCV, pages 622-631, 2019.","title":"Feature weighting and boosting for few-shot segmentation","authors":"Todorovic Nguyen, Khoi Nguyen, Sinisa Todorovic","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qiao et al., 2019] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In CVPR, pages 1505-1514, 2019.","title":"Mirrorgan: Learning text-to-image generation by redescription","authors":"Tingting Qiao, Jing Zhang, Duanqing Xu, Dacheng Tao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rakelly et al., 2018] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A Efros, and Sergey Levine. Few-shot segmentation propagation with guided networks. arXiv preprint arXiv:1806.07373, 2018.","title":"Few-shot segmentation propagation with guided networks","authors":"Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A Efros, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rusu et al., 2018] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent em- bedding optimization. arXiv preprint arXiv:1807.05960, 2018.","title":"Meta-learning with latent embedding optimization","authors":"Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Santoro et al., 2016] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lilli- crap. One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065, 2016.","title":"One-shot learning with memory-augmented neural networks","authors":"Santoro","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shaban et al., 2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410, 2017.","title":"One-shot learning for semantic segmentation","authors":"Shaban","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Ad- vances in Neural Information Processing Systems, pages 4077-4087, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sung et al., 2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learn- ing. In CVPR, pages 1199-1208, 2018.","title":"Learning to compare: Relation network for few-shot learning","authors":"Sung","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tian et al., 2019] Pinzhuo Tian, Zhangkai Wu, Lei Qi, Lei Wang, Yinghuan Shi, and Yang Gao. Differentiable meta- learning model for few-shot semantic segmentation. arXiv preprint arXiv:1911.10371, 2019.","title":"Differentiable metalearning model for few-shot semantic segmentation","authors":"Tian","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot im- age semantic segmentation with prototype alignment. In ICCV, pages 9197-9206, 2019.","title":"Panet: Few-shot image semantic segmentation with prototype alignment","authors":"Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2012] Hanwang Zhang, Zheng-Jun Zha, Shuicheng Yan, Jingwen Bian, and Tat-Seng Chua. Attribute feedback. In MM, pages 79-88, 2012.","title":"Attribute feedback","authors":"Hanwang Zhang, Zheng-Jun Zha, Shuicheng Yan, Jingwen Bian, Tat-Seng Chua","year":"2012","venue":"MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2014] Hanwang Zhang, Zheng-Jun Zha, Yang Yang, Shuicheng Yan, and Tat-Seng Chua. Robust (semi) nonnegative graph embedding. IEEE transactions on im- age processing, 23(7):2996-3012, 2014.","title":"Robust (semi) nonnegative graph embedding","authors":"Hanwang Zhang, Zheng-Jun Zha, Yang Yang, Shuicheng Yan, Tat-Seng Chua","year":"2014","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas Huang. Sg-one: Similarity guidance net- work for one-shot semantic segmentation. arXiv preprint arXiv:1810.09091, 2018.","title":"Sg-one: Similarity guidance network for one-shot semantic segmentation","authors":"Xiaolin Zhang, Yunchao Wei, Yi Yang, Thomas Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019a] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic seg- mentation networks with iterative refinement and attentive few-shot learning. In CVPR, pages 5217-5226, 2019.","title":"Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning","authors":"Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019b] Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. Category anchor-guided unsupervised domain adaptation for semantic segmentation. In Ad- vances in Neural Information Processing Systems, pages 433-443, 2019.","title":"Category anchor-guided unsupervised domain adaptation for semantic segmentation","authors":"Zhang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiao- juan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 2881-2890, 2017.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2019] Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In ICCV, pages 593-602, 2019.","title":"Asymmetric non-local neural networks for semantic segmentation","authors":"Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, Xiang Bai","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}