{"type":"ScholarlyArticle","title":"WiCo: Win-win Cooperation of Bottom-up and Top-down Referring Image Segmentation","authors":[{"name":"Zesen Cheng","slug":"zesen-cheng","given":"Zesen","family":"Cheng","affiliation":"School of Electronic and Computer Engineering , Peking University , Shenzhen , China; AI for Science (AI4S)-Preferred Program , Peking University Shenzhen Graduate School , China"},{"name":"Peng Jin","slug":"peng-jin","given":"Peng","family":"Jin","affiliation":"School of Electronic and Computer Engineering , Peking University , Shenzhen , China; AI for Science (AI4S)-Preferred Program , Peking University Shenzhen Graduate School , China"},{"name":"Hao Li","slug":"hao-li","given":"Hao","family":"Li","affiliation":"School of Electronic and Computer Engineering , Peking University , Shenzhen , China; AI for Science (AI4S)-Preferred Program , Peking University Shenzhen Graduate School , China"},{"name":"Kehan Li","slug":"kehan-li","given":"Kehan","family":"Li","affiliation":"School of Electronic and Computer Engineering , Peking University , Shenzhen , China; AI for Science (AI4S)-Preferred Program , Peking University Shenzhen Graduate School , China"},{"name":"Siheng Li","slug":"siheng-li","given":"Siheng","family":"Li","affiliation":"Tsinghua University , Beijing , China"},{"name":"Xiangyang Ji","slug":"xiangyang-ji","given":"Xiangyang","family":"Ji","affiliation":"Tsinghua University , Beijing , China"},{"name":"Chang Liu","slug":"chang-liu","given":"Chang","family":"Liu","affiliation":"Tsinghua University , Beijing , China"},{"name":"Jie Chen","slug":"jie-chen","given":"Jie","family":"Chen","affiliation":"School of Electronic and Computer Engineering , Peking University , Shenzhen , China; AI for Science (AI4S)-Preferred Program , Peking University Shenzhen Graduate School , China; Peng Cheng Laboratory , Shenzhen , China"}],"abstract":"The top-down and bottom-up methods are two mainstreams of referring segmentation, while both methods have their own intrinsic weaknesses. Top-down methods are chiefly disturbed by Polar Negative (PN) errors owing to the lack of fine-grained cross-modal alignment. Bottom-up methods are mainly perturbed by Inferior Positive (IP) errors due to the lack of prior object information. Nevertheless, we discover that two types of methods are highly complementary for restraining respective weaknesses but the direct average combination leads to harmful interference. In this context, we build Win-win Cooperation (WiCo) to exploit complementary nature of two types of methods on both interaction and integration aspects for achieving a win-win improvement. For the interaction aspect, Complementary Feature Interaction (CFI) introduces prior object information to bottom-up branch and provides fine-grained information to top-down branch for complementary feature enhancement. For the integration aspect, Gaussian Scoring Integration (GSI) models the gaussian performance distributions of two branches and weighted integrates results by sampling confident scores from the distributions. With our WiCo, several prominent bottom-up and top-down combinations achieve remarkable improvements on three common datasets with reasonable extra costs, which justifies effectiveness and generality of our method.","doi":"10.24963/ijcai.2023/71","url":"https://ijcai.org/proceedings/2023/71","pdf":"https://ijcai.org/proceedings/2023/0071.pdf","markdown":"https://ijcai.org/proceedings/2023/71.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/71","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":71,"pages":{"first":636,"last":644},"keywords":["Computer Vision: CV: Vision and language","Computer Vision: CV: Segmentation"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2019a] Ding-Jie Chen, Songhao Jia, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu. See-through- text grouping for referring image segmentation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 7454-7463, 2019.","title":"See-throughtext grouping for referring image segmentation","authors":"Chen","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019b] Yi-Wen Chen, Yi-Hsuan Tsai, Tiantian Wang, Yen-Yu Lin, and Ming-Hsuan Yang. Referring ex- pression object segmentation with caption-aware consis- tency. arXiv preprint arXiv:1910.04748, 2019.","title":"Referring expression object segmentation with caption-aware consistency","authors":"Chen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2021] Bowen Cheng, Ishan Misra, Alexan- der G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. arXiv preprint arXiv:2112.01527, 2021.","title":"Masked-attention mask transformer for universal image segmentation","authors":"Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, Rohit Girdhar","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2021] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 16321-16330, 2021.","title":"Vision-language transformer and query generation for referring segmentation","authors":"Ding","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng et al., 2021] Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with co- attention embedding for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15506-15515, 2021.","title":"Encoder fusion network with coattention embedding for referring image segmentation","authors":"Guang Feng, Zhiwei Hu, Lihe Zhang, Huchuan Lu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2016a] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language ex- pressions. In European Conference on Computer Vision, pages 108-124. Springer, 2016.","title":"Segmentation from natural language expressions","authors":"Hu","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2016b] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4555-4564, 2016.","title":"Natural language object retrieval","authors":"Hu","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2020] Shaofei Huang, Tianrui Hui, Si Liu, Guanbin Li, Yunchao Wei, Jizhong Han, Luoqi Liu, and Bo Li. Referring image segmentation via cross-modal pro- gressive comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10488-10497, 2020.","title":"Referring image segmentation via cross-modal progressive comprehension","authors":"Huang","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hui et al., 2020] Tianrui Hui, Si Liu, Shaofei Huang, Guan- bin Li, Sansi Yu, Faxi Zhang, and Jizhong Han. Linguistic structure guided context modeling for referring image seg- mentation. In European Conference on Computer Vision, pages 59-75. Springer, 2020.","title":"Linguistic structure guided context modeling for referring image segmentation","authors":"Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, Jizhong Han","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jing et al., 2021] Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tieniu Tan. Locate then segment: A strong pipeline for referring image segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9858-9867, 2021.","title":"Locate then segment: A strong pipeline for referring image segmentation","authors":"Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, Tieniu Tan","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2022] Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution- free referring image segmentation using transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18145-18154, 2022.","title":"Restr: Convolutionfree referring image segmentation using transformers","authors":"Kim","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Sigal, 2021] Muchen Li and Leonid Sigal. Refer- ring transformer: A one-step approach to multi-task visual grounding. Advances in Neural Information Processing Systems, 34:19652-19664, 2021.","title":"Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding","authors":"Sigal Li","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refine- ment networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5745- 5753, 2018.","title":"Referring image segmentation via recurrent refinement networks","authors":"Li","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree networks for visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4673-4682, 2019.","title":"Learning to assemble neural module tree networks for visual grounding","authors":"Daqing Liu, Hanwang Zhang, Feng Wu, Zheng-Jun Zha","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo et al., 2020a] Gen Luo, Yiyi Zhou, Rongrong Ji, Xi- aoshuai Sun, Jinsong Su, Chia-Wen Lin, and Qi Tian. Cas- cade grouped attention network for referring expression segmentation. In Proceedings of the 28th ACM Interna- tional Conference on Multimedia, pages 1274-1282, 2020.","title":"Cascade grouped attention network for referring expression segmentation","authors":"Luo","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2020b] Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Li- ujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expres- sion comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 10034-10043, 2020.","title":"Multi-task collaborative network for joint referring expression comprehension and segmentation","authors":"Luo","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mao et al., 2016] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Mur- phy. Generation and comprehension of unambiguous ob- ject descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-20, 2016.","title":"Generation and comprehension of unambiguous object descriptions","authors":"Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, Kevin Murphy","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2018] Hengcan Shi, Hongliang Li, Fanman Meng, and Qingbo Wu. Key-word-aware network for re- ferring expression image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 38-54, 2018.","title":"Key-word-aware network for referring expression image segmentation","authors":"Shi","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van den Oord et al., 2017] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete represen- tation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30.","title":"Vinyals, and koray kavukcuoglu. Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2022] Zhaoqing Wang, Yu Lu, Qiang Li, Xun- qiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11686-11695, 2022.","title":"Cris: Clip-driven referring image segmentation","authors":"Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, Tongliang Liu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2021] Sibei Yang, Meng Xia, Guanbin Li, Hong-Yu Zhou, and Yizhou Yu. Bottom-up shift and rea- soning for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11266-11275, 2021.","title":"Bottom-up shift and reasoning for referring image segmentation","authors":"Sibei Yang, Meng Xia, Guanbin Li, Hong-Yu Zhou, Yizhou Yu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2022] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 18155-18165, 2022.","title":"Lavt: Language-aware vision transformer for referring image segmentation","authors":"Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, Philip Hs Torr","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ye et al., 2019] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for refer- ring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10502-10511, 2019.","title":"Cross-modal self-attention network for referring image segmentation","authors":"Ye","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2016] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling con- text in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer, 2016.","title":"Modeling context in referring expressions","authors":"Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, Tamara L Berg","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2018] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307- 1315, 2018.","title":"Mattnet: Modular attention network for referring expression comprehension","authors":"Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L Berg","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2022] Zicheng Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, and Wei Ke. Coupalign: Coupling word-pixel with sentence-mask alignments for referring image segmentation. arXiv preprint arXiv:2212.01769, 2022.","title":"Coupalign: Coupling word-pixel with sentence-mask alignments for referring image segmentation","authors":"Zicheng Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, Wei Ke","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2022] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. arXiv preprint arXiv:2203.16265, 2022.","title":"Seqtr: A simple yet universal network for visual grounding","authors":"Zhu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}