{"type":"ScholarlyArticle","title":"SimPropNet: Improved Similarity Propagation for Few-shot Image Segmentation","authors":[{"name":"Siddhartha Gairola","slug":"siddhartha-gairola","given":"Siddhartha","family":"Gairola","affiliation":"IIIT Hyderabad"},{"name":"Mayur Hemani","slug":"mayur-hemani","given":"Mayur","family":"Hemani","affiliation":"Media and Data Science Research Lab , Adobe Experience Cloud"},{"name":"Ayush Chopra","slug":"ayush-chopra","given":"Ayush","family":"Chopra","affiliation":"Media and Data Science Research Lab , Adobe Experience Cloud"},{"name":"Balaji Krishnamurthy","slug":"balaji-krishnamurthy","given":"Balaji","family":"Krishnamurthy","affiliation":"Media and Data Science Research Lab , Adobe Experience Cloud"}],"abstract":"Few-shot segmentation (FSS) methods perform image segmentation for a particular object class in a target (query) image, using a small set of (support) image-mask pairs. Recent deep neural network based FSS methods leverage high-dimensional feature similarity between the foreground features of the support images and the query image features. In this work, we demonstrate gaps in the utilization of this similarity information in existing methods, and present a framework - SimPropNet, to bridge those gaps. We propose to jointly predict the support and query masks to force the support features to share characteristics with the query features. We also propose to utilize similarities in the background regions of the query and support images using a novel foreground-background attentive fusion mechanism. Our method achieves state-of-the-art results for one-shot and five-shot segmentation on the PASCAL-5i dataset. The paper includes detailed analysis and ablation studies for the proposed improvements and quantitative comparisons with contemporary methods.","doi":"10.24963/ijcai.2020/80","url":"https://ijcai.org/proceedings/2020/80","pdf":"https://ijcai.org/proceedings/2020/0080.pdf","markdown":"https://ijcai.org/proceedings/2020/80.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/80","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":80,"pages":{"first":573,"last":579},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning: Convolutional networks","Machine Learning: Semi-Supervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24830/file_24830.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"Yagız Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Polle- feys, and Wojciech Matusik. Semantic soft segmenta- tion. ACM Trans. Graph. (Proc. SIGGRAPH), 37(4):72:1- 72:13, 2018.","title":"Semantic soft segmentation","authors":"Yagız Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys, Wojciech Matusik","year":"2018","venue":"Proc. SIGGRAPH)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:2481- 2495, 2017.","title":"Segnet: A deep convolutional encoder-decoder architecture for image segmentation","authors":"Vijay Badrinarayanan, Alex Kendall, Roberto Cipolla","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Liang-Chieh Chen, George Papandreou, Iasonas Kokki- nos, Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with deep convolutional nets and fully con- nected crfs. CoRR, abs/1412.7062, 2014.","title":"Semantic image segmentation with deep convolutional nets and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Liang-Chieh Chen, George Papandreou, Iasonas Kokki- nos, Kevin Murphy, and Alan L. Yuille. Deeplab: Se- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 40:834-848, 2016.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for se- mantic image segmentation. ArXiv, abs/1706.05587, 2017.","title":"Rethinking atrous convolution for semantic image segmentation","authors":"Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Liang-Chieh Chen, Yukun Zhu, George Papandreou, Flo- rian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmen- tation. In ECCV, 2018.","title":"Encoder-decoder with atrous separable convolution for semantic image segmentation","authors":"Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Jose Dolz, Karthik Gopinath, Jing Yuan, Herve Lom- baert, Christian Desrosiers, and Ismail Ben Ayed. Hyperdense-net: A hyper-densely connected cnn for multi-modal image segmentation. IEEE Transactions on Medical Imaging, 38:1116-1126, 2018.","title":"Hyperdense-net: A hyper-densely connected cnn for multi-modal image segmentation","authors":"Jose Dolz, Karthik Gopinath, Jing Yuan, Herve Lombaert, Christian Desrosiers, Ismail Ben, Ayed","year":"2018","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Nanqing Dong and Eric P. Xing. Few-shot semantic seg- mentation with prototype learning. In BMVC, 2018.","title":"Few-shot semantic segmentation with prototype learning","authors":"Nanqing Dong, Eric P Xing","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.","title":"The PASCAL Visual Object Classes Challenge","authors":"M Everingham, L Van Gool, C K I Williams, J Winn, A Zisserman","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A. Efros, and Sergey Levine. Conditional networks for few-shot semantic segmentation. In ICLR, 2018.","title":"Conditional networks for few-shot semantic segmentation","authors":"Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A Efros, Sergey Levine","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image seg- mentation. CoRR, abs/1505.04597, 2015.","title":"U-net: Convolutional networks for biomedical image segmentation","authors":"Olaf Ronneberger, Philipp Fischer, Thomas Brox","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexan- der C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Com- puter Vision (IJCV), 115(3):211-252, 2015.","title":"ImageNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for seman- tic segmentation. In Gabriel Brostow Tae-Kyun Kim, Stefanos Zafeiriou and Krystian Mikolajczyk, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 167.1-167.13. BMVA Press, September 2017.","title":"One-shot learning for semantic segmentation","authors":"Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, Byron Boots","year":"2017","venue":"Proceedings of the British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Mennatullah Siam and Boris N. Oreshkin. Adap- tive masked weight imprinting for few-shot segmentation. CoRR, abs/1902.11123, 2019.","title":"Adaptive masked weight imprinting for few-shot segmentation","authors":"Mennatullah Siam, Boris N Oreshkin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representa- tions, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"3rd International Conference on Learning Representations, ICLR 2015","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Ashish Sinha and José Estevan Dolz. Multi-scale guided attention for medical image segmentation. ArXiv, abs/1906.02849, 2019.","title":"Multi-scale guided attention for medical image segmentation","authors":"Ashish Sinha, José Estevan, Dolz","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Jake Snell, Kevin Swersky, and Richard Zemel. Proto- typical networks for few-shot learning. 03 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. Soft color segmentation and its applications. IEEE Trans. Pattern Anal. Mach. Intell., 29(9), September 2007.","title":"Soft color segmentation and its applications","authors":"Yu-Wing Tai, Jiaya Jia, Chi-Keung Tang","year":"2007","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Jianchao Tan, Jyh-Ming Lien, and Yotam Gingold. De- composing images into layers via rgb-space geometry. ACM Trans. Graph., 36(1), November 2016.","title":"Decomposing images into layers via rgb-space geometry","authors":"Jianchao Tan, Jyh-Ming Lien, Yotam Gingold","year":"2016","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lem- pitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/1607.08022, 2016.","title":"Instance normalization: The missing ingredient for fast stylization","authors":"Dmitry Ulyanov, Andrea Vedaldi, Victor S Lempitsky","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, D. Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recogni- tion. ArXiv, abs/1908.07919, 2019.","title":"Deep high-resolution representation learning for visual recognition","authors":"Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, D Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, Bin Xiao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image se- mantic segmentation with prototype alignment. ArXiv, abs/1908.06391, 2019.","title":"Panet: Few-shot image semantic segmentation with prototype alignment","authors":"Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. 10 2018.","title":"Sg-one: Similarity guidance network for one-shot semantic segmentation","authors":"Xiaolin Zhang, Yunchao Wei, Yi Yang, Thomas Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmentation net- works with iterative refinement and attentive few-shot learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.","title":"Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning","authors":"Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, Chunhua Shen","year":"2019","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiao- gang Wang, and Jiaya Jia. Pyramid scene parsing network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6230-6239, 2016.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}