{"type":"ScholarlyArticle","title":"Weakly Supervised Few-shot Object Segmentation using Co-Attention with Visual and Semantic Embeddings","authors":[{"name":"Mennatullah Siam","slug":"mennatullah-siam","given":"Mennatullah","family":"Siam","affiliation":"University of Alberta; HiSilicon , Huawei Research"},{"name":"Naren Doraiswamy","slug":"naren-doraiswamy","given":"Naren","family":"Doraiswamy","affiliation":"Indian Institute of Science"},{"name":"Boris N. Oreshkin","slug":"boris-n-oreshkin","given":"Boris N.","family":"Oreshkin","affiliation":"Element AI"},{"name":"Hengshuai Yao","slug":"hengshuai-yao","given":"Hengshuai","family":"Yao","affiliation":"HiSilicon , Huawei Research"},{"name":"Martin Jagersand","slug":"martin-jagersand","given":"Martin","family":"Jagersand","affiliation":"University of Alberta"}],"abstract":"Significant progress has been made recently in developing few-shot object segmentation methods. Learning is shown to be successful in few-shot segmentation settings, using pixel-level, scribbles and bounding box supervision. This paper takes another approach, i.e., only requiring image-level label for few-shot object segmentation. We propose a novel multi-modal interaction module for few-shot object segmentation that utilizes a co-attention mechanism using both visual and word embedding. Our model using image-level labels achieves 4.8% improvement over previously proposed image-level few-shot object segmentation. It also outperforms state-of-the-art methods that use weak bounding box supervision on PASCAL-5^i. Our results show that few-shot segmentation benefits from utilizing word embeddings, and that we are able to perform few-shot segmentation using stacked joint visual semantic processing with weak image-level labels. We further propose a novel setup, Temporal Object Segmentation for Few-shot Learning (TOSFL) for videos. TOSFL can be used on a variety of public video data such as Youtube-VOS, as demonstrated in both instance-level and category-level TOSFL experiments.","doi":"10.24963/ijcai.2020/120","url":"https://ijcai.org/proceedings/2020/120","pdf":"https://ijcai.org/proceedings/2020/0120.pdf","markdown":"https://ijcai.org/proceedings/2020/120.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/120","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":120,"pages":{"first":860,"last":867},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning","Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26687/file_26687.m3u8","length_sec":569,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017a] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intel- ligence, 40(4):834-848, 2017.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Chen","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017b] Liang-Chieh Chen, George Papan- dreou, Florian Schroff, and Hartwig Adam. Rethink- ing atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.","title":"Rethinking atrous convolution for semantic image segmentation","authors":"Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Se- bastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene under- standing. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 3213-3223, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dehghan et al., 2019] Masood Dehghan, Zichen Zhang, Mennatullah Siam, Jun Jin, Laura Petrich, and Mar- tin Jagersand. Online object and task learning via hu- man robot interaction. In 2019 International Conference on Robotics and Automation (ICRA), pages 2132-2138. IEEE, 2019.","title":"Online object and task learning via human robot interaction","authors":"Masood Dehghan, Zichen Zhang, Mennatullah Siam, Jun Jin, Laura Petrich, Martin Jagersand","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong and Xing, 2018] Nanqing Dong and Eric P. Xing. Few-shot semantic segmentation with prototype learning. In BMVC, volume 3, page 4, 2018.","title":"Few-shot semantic segmentation with prototype learning","authors":"Dong, Xing ;, Nanqing Dong, Eric P Xing","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Everingham et al., 2015] Mark Everingham, S.M. Ali Es- lami, Luc Van Gool, Christopher K.I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-136, 2015.","title":"The pascal visual object classes challenge: A retrospective","authors":"S M Mark Everingham, Luc Ali Eslami, Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2015","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hsieh et al., 2019] Ting-I Hsieh, Yi-Chen Lo, Hwann- Tzong Chen, and Tyng-Luh Liu. One-shot object detection with co-attention and co-excitation. In Advances in Neural Information Processing Systems, pages 2721-2730, 2019.","title":"One-shot object detection with co-attention and co-excitation","authors":"Ting-I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, Tyng-Luh Liu","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Infor- mation Processing Systems, pages 289-297, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2019] Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3623-3632, 2019.","title":"See more, know more: Unsupervised video object segmentation with co-attention siamese networks","authors":"Lu","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pirk et al., 2019] Sören Pirk, Mohi Khansari, Yunfei Bai, Corey Lynch, and Pierre Sermanet. Online object rep- resentations with contrastive learning. arXiv preprint arXiv:1906.04312, 2019.","title":"Online object representations with contrastive learning","authors":"Sören Pirk, Mohi Khansari, Yunfei Bai, Corey Lynch, Pierre Sermanet","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rakelly et al., 2018] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine. Conditional networks for few-shot semantic segmentation. 2018.","title":"Conditional networks for few-shot semantic segmentation","authors":"Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Raza et al., 2019] Hasnain Raza, Mahdyar Ravanbakhsh, Tassilo Klein, and Moin Nabi. Weakly supervised one shot segmentation. In Proceedings of the IEEE Interna- tional Conference on Computer Vision Workshops, pages 0-0, 2019.","title":"Weakly supervised one shot segmentation","authors":"Hasnain Raza, Mahdyar Ravanbakhsh, Tassilo Klein, Moin Nabi","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shaban et al., 2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410, 2017.","title":"One-shot learning for semantic segmentation","authors":"Shaban","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Siam et al., 2019] Mennatullah Siam, Boris N Oreshkin, and Martin Jagersand. Amp: Adaptive masked proxies for few-shot segmentation. In Proceedings of the IEEE In- ternational Conference on Computer Vision, pages 5249- 5258, 2019.","title":"Amp: Adaptive masked proxies for few-shot segmentation","authors":"Siam","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2019] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE International Conference on Computer Vision, pages 9197-9206, 2019.","title":"Panet: Few-shot image semantic segmentation with prototype alignment","authors":"Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2018] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.","title":"Youtube-vos: A large-scale video object segmentation benchmark","authors":"Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, Thomas Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 21-29, 2016.","title":"Stacked attention networks for image question answering","authors":"Yang","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2019a] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region- based one-shot semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 9587-9595, 2019.","title":"Pyramid graph networks with connection attentions for regionbased one-shot semantic segmentation","authors":"Zhang","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2019b] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic seg- mentation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5217- 5226, 2019.","title":"Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning","authors":"Zhang","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}