{"type":"ScholarlyArticle","title":"One Weird Trick to Improve Your Semi-Weakly Supervised Semantic Segmentation Model","authors":[{"name":"Wonho Bae","slug":"wonho-bae","given":"Wonho","family":"Bae","affiliation":"University of British Columbia"},{"name":"Junhyug Noh","slug":"junhyug-noh","given":"Junhyug","family":"Noh","affiliation":"Lawrence Livermore National Laboratory"},{"name":"Milad Jalali Asadabadi","slug":"milad-jalali-asadabadi","given":"Milad","family":"Jalali Asadabadi","affiliation":"University of British Columbia"},{"name":"Danica J. Sutherland","slug":"danica-j-sutherland","given":"Danica J.","family":"Sutherland","affiliation":"University of British Columbia; Alberta Machine Intelligence Institute"}],"abstract":"Semi-weakly supervised semantic segmentation (SWSSS) aims to train a model to identify objects in images based on a small number of images with pixel-level labels, and many more images with only image-level labels. Most existing SWSSS algorithms extract pixel-level pseudo-labels from an image classifier - a very difficult task to do well, hence requiring complicated architectures and extensive hyperparameter tuning on fully-supervised validation sets. We propose a method called prediction filtering, which instead of extracting pseudo-labels, just uses the classifier as a classifier: it ignores any segmentation predictions from classes which the classifier is confident are not present. Adding this simple post-processing method to baselines gives results competitive with or better than prior SWSSS algorithms. Moreover, it is compatible with pseudo-label methods: adding prediction filtering to existing SWSSS algorithms further improves segmentation performance.","doi":"10.24963/ijcai.2022/389","url":"https://ijcai.org/proceedings/2022/389","pdf":"https://ijcai.org/proceedings/2022/0389.pdf","markdown":"https://ijcai.org/proceedings/2022/389.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/389","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":389,"pages":{"first":2805,"last":2811},"keywords":["Machine Learning: Weakly Supervised Learning","Computer Vision: Segmentation","Machine Learning: Semi-Supervised Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986448","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984591","length_sec":355,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahn and Kwak, 2018] J. Ahn and S. Kwak. Learning pixel- level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation. In CVPR, 2018.","title":"Learning pixellevel Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation","authors":"Ahn, Kwak, ] J Ahn, S Kwak","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2015] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic Image Segmenta- tion with Deep Convolutional Nets and Fully Connected CRFs. In ICLR, 2015.","title":"Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs","authors":"L.-C Chen, G Papandreou, I Kokkinos, K Murphy, A L Yuille","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking Atrous Convolution for Seman- tic Image Segmentation. In CVPR, 2017.","title":"Rethinking Atrous Convolution for Semantic Image Segmentation","authors":"L.-C Chen, G Papandreou, F Schroff, H Adam","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] L. Chen, W. Wu, C. Fu, X. Han, and Y. Zhang. Weakly Supervised Semantic Segmentation with Boundary Exploration. In ECCV, 2020.","title":"Weakly Supervised Semantic Segmentation with Boundary Exploration","authors":"L Chen, W Wu, C Fu, X Han, Y Zhang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choe and Shim, 2019] J. Choe and H. Shim. Attention- Based Dropout Layer for Weakly Supervised Object Lo- calization. In CVPR, 2019.","title":"Attention-Based Dropout Layer for Weakly Supervised Object Localization","authors":"Shim Choe, ] J Choe, H Shim","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choe et al., 2020] J. Choe, S. Oh, S. Lee, S. Chun, Z. Akata, and H. Shim. Evaluating Weakly Supervised Object Local- ization Methods Right. In CVPR, 2020.","title":"Evaluating Weakly Supervised Object Localization Methods Right","authors":"J Choe, S Oh, S Lee, S Chun, Z Akata, H Shim","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dang et al., 2022] V. Dang, F. Galati, R. Cortese, G. Di Gi- acomo, V. Marconetto, P. Mathur, K. Lekadir, M. Lorenzi, F. Prados, and M. Zuluaga. Vessel-CAPTCHA: An Effi- cient Learning Framework for Vessel Annotation and Seg- mentation. Medical Image Analysis, 2022.","title":"Vessel-CAPTCHA: An Efficient Learning Framework for Vessel Annotation and Segmentation","authors":"V Dang, F Galati, R Cortese, G Di Giacomo, V Marconetto, P Mathur, K Lekadir, M Lorenzi, F Prados, M Zuluaga","year":"2022","venue":"Medical Image Analysis","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Everingham et al., 2015] M. Everingham, A. Eslami, L. Gool, C. Williams, J. Winn, and A. Zisserman. Pascal Vis. Obj. Class Challenge: A Retrospective. IJCV, 2015.","title":"Class Challenge: A Retrospective","authors":"M Everingham, A Eslami, L Gool, C Williams, J Winn, A Zisserman, Vis, Obj","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] K He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"X He, S Zhang, J Ren, Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2018] Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang. Weakly-supervised Semantic Segmentation Network with Deep Seeded Region Growing. In CVPR, 2018.","title":"Weakly-supervised Semantic Segmentation Network with Deep Seeded Region Growing","authors":"Z Huang, X Wang, J Wang, W Liu, J Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2019] P.-T. Jiang, Q. Hou, Y. Cao, M.-M. Cheng, Y. Wei, and H.-K. Xiong. Integral Object Mining via Online Attention Accumulation. In ICCV, 2019.","title":"Integral Object Mining via Online Attention Accumulation","authors":"P.-T Jiang, Q Hou, Y Cao, M.-M Cheng, Y Wei, H.-K Xiong","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krähenbühl and Koltun, 2011] P. Krähenbühl and V. Koltun. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. In NIPS, 2011.","title":"Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials","authors":"P Krähenbühl, V Koltun","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lai et al., 2021] X. Lai, Z. Tian, L. Jiang, S. Liu, H. Zhao, L. Wang, and J. Jia. Semi-Supervised Semantic Segmen- tation with Directional Context-aware Consistency. In CVPR, 2021.","title":"Semi-Supervised Semantic Segmentation with Directional Context-aware Consistency","authors":"Lai","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2019] J. Lee, E. Kim, S. Lee, J. Lee, and S. Yoon. Ficklenet: Weakly and Semi-supervised Semantic Image Segmentation Using Stochastic Inference. In CVPR, 2019.","title":"Ficklenet: Weakly and Semi-supervised Semantic Image Segmentation Using Stochastic Inference","authors":"J Lee, E Kim, S Lee, J Lee, S Yoon","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2021] J. Lee, E. Kim, and S. Yoon. Anti- Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation. In CVPR, 2021.","title":"Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation","authors":"J Lee, E Kim, S Yoon","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018] K. Li, Z. Wu, K. Peng, J. Ernst, and Y. Fu. Tell Me Where to Look: Guided Attention Inference Net- work. In CVPR, 2018.","title":"Tell Me Where to Look: Guided Attention Inference Network","authors":"K Li, Z Wu, K Peng, J Ernst, Y Fu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2014] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Zitnick. MS COCO: Common Objects in Context. In ECCV, 2014.","title":"MS COCO: Common Objects in Context","authors":"T Lin, M Maire, S Belongie, J Hays, P Perona, D Ramanan, P Dollár, C Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo and Yang, 2020] W. Luo and M. Yang. Semi- supervised Semantic Segmentation via Strong-weak Dual- branch Network. In ECCV, 2020.","title":"Semisupervised Semantic Segmentation via Strong-weak Dualbranch Network","authors":"W Luo, M Yang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ouali et al., 2020] Y. Ouali, C. Hudelot, and M. Tami. Semi-supervised Semantic Segmentation with Cross- consistency Training. In CVPR, 2020.","title":"Semi-supervised Semantic Segmentation with Crossconsistency Training","authors":"Y Ouali, C Hudelot, M Tami","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papandreou et al., 2015] G. Papandreou, L. Chen, K. P Murphy, and A. L Yuille. Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation. In ICCV, 2015.","title":"Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation","authors":"G Papandreou, L Chen, K P Murphy, A L Yuille","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Selvaraju et al., 2017] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-Cam: Visual Explanations From Deep Networks via Gradient- Based Localization. In ICCV, 2017.","title":"Grad-Cam: Visual Explanations From Deep Networks via Gradient-Based Localization","authors":"R R Selvaraju, M Cogswell, A Das, R Vedantam, D Parikh, D Batra","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2015] K. Simonyan and A. Zis- serman. Very Deep Convolutional Networks for Large- Scale Image Recognition. In ICLR, 2015.","title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","authors":"K Simonyan, A Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Singh and Lee, 2017] K. Singh and Y. Lee. Hide-And-Seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In ICCV, 2017.","title":"Hide-And-Seek: Forcing a network to be meticulous for weakly-supervised object and action localization","authors":"K Singh, Y Lee","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Souly et al., 2017] N. Souly, C. Spampinato, and M. Shah. Semi and Weakly Supervised Semantic Segmentation Us- ing Generative Adversarial Network. ICCV, 2017.","title":"Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network","authors":"N Souly, C Spampinato, M Shah","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Strudel et al., 2021] R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for Semantic Seg- mentation. In ICCV, 2021.","title":"Segmenter: Transformer for Semantic Segmentation","authors":"R Strudel, R Garcia, I Laptev, C Schmid","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2020] G. Sun, W. Wang, J. Dai, and L. Van Gool. Mining Cross-image Semantics for Weakly Supervised Se- mantic Segmentation. In ECCV, 2020.","title":"Mining Cross-image Semantics for Weakly Supervised Semantic Segmentation","authors":"G Sun, W Wang, J Dai, L Van Gool","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2020] Y. Wang, J. Zhang, M. Kan, S. Shan, and X. Chen. Self-supervised Equivariant Attention Mecha- nism for Weakly Supervised Semantic Segmentation. In CVPR, 2020.","title":"Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation","authors":"Y Wang, J Zhang, M Kan, S Shan, X Chen","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2021] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, W. Liu, and B. Xiao. Deep High-Resolution Representa- tion Learning for Visual Recognition. IEEE T-PAMI, 2021.","title":"Deep High-Resolution Representation Learning for Visual Recognition","authors":"J Wang, K Sun, T Cheng, B Jiang, C Deng, Y Zhao, D Liu, Y Mu, M Tan, X Wang, W Liu, B Xiao","year":"2021","venue":"IEEE T-PAMI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wei et al., 2018] Y. Wei, H. Xiao, H. Shi, Z. Jie, J. Feng, and T. S Huang. Revisiting Dilated Convolution: A Sim- ple Approach for Weakly-and Semi-supervised Semantic Segmentation. In CVPR, 2018.","title":"Revisiting Dilated Convolution: A Simple Approach for Weakly-and Semi-supervised Semantic Segmentation","authors":"Y Wei, H Xiao, H Shi, Z Jie, J Feng, T S Huang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu and Koltun, 2016] F. Yu and V. Koltun. Multi-scale Con- text Aggregation by Dilated Convolutions. In ICLR, 2016.","title":"Multi-scale Context Aggregation by Dilated Convolutions","authors":"F Yu, V Koltun","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2018] X. Zhang, Y. Wei, J. Feng, and T. S. Yang, Y. Huang. Adversarial Complementary Learning for Weakly Supervised Object Localization. In CVPR, 2018.","title":"Adversarial Complementary Learning for Weakly Supervised Object Localization","authors":"Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2016] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. In CVPR, 2016.","title":"Learning Deep Features for Discriminative Localization","authors":"B Zhou, A Khosla, A Lapedriza, A Oliva, A Torralba","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}