{"type":"ScholarlyArticle","title":"Self-supervised Semantic Segmentation Grounded in Visual Concepts","authors":[{"name":"Wenbin He","slug":"wenbin-he","given":"Wenbin","family":"He","affiliation":"Robert Bosch Research and Technology Center North America"},{"name":"William Surmeier","slug":"william-surmeier","given":"William","family":"Surmeier","affiliation":"Robert Bosch Research and Technology Center North America"},{"name":"Arvind Kumar Shekar","slug":"arvind-kumar-shekar","given":"Arvind Kumar","family":"Shekar","affiliation":"Robert Bosch GmbH"},{"name":"Liang Gou","slug":"liang-gou","given":"Liang","family":"Gou","affiliation":"Robert Bosch Research and Technology Center North America"},{"name":"Liu Ren","slug":"liu-ren","given":"Liu","family":"Ren","affiliation":"Robert Bosch Research and Technology Center North America"}],"abstract":"Unsupervised semantic segmentation requires assigning a label to every pixel without any human annotations.  Despite recent advances in self-supervised representation learning for individual images, unsupervised semantic segmentation with pixel-level representations is still a challenging task and remains underexplored.  In this work, we propose a self-supervised pixel representation learning method for semantic segmentation by using visual concepts (i.e., groups of pixels with semantic meanings, such as parts, objects, and scenes) extracted from images.  To guide self-supervised learning, we leverage three types of relationships between pixels and concepts, including the relationships between pixels and local concepts, local and global concepts, as well as the co-occurrence of concepts.  We evaluate the learned pixel embeddings and visual concepts on three datasets, including PASCAL VOC 2012, COCO 2017, and DAVIS 2017.  Our results show that the proposed method gains consistent and substantial improvements over recent unsupervised semantic segmentation approaches, and also demonstrate that visual concepts can reveal insights into image datasets.","doi":"10.24963/ijcai.2022/133","url":"https://ijcai.org/proceedings/2022/133","pdf":"https://ijcai.org/proceedings/2022/0133.pdf","markdown":"https://ijcai.org/proceedings/2022/133.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/133","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":133,"pages":{"first":949,"last":955},"keywords":["AI Ethics, Trust, Fairness: Explainability and Interpretability","Machine Learning: Self-supervised Learning","Computer Vision: Segmentation","Computer Vision: Interpretability and Transparency","Computer Vision: Representation Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986928","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985071","length_sec":331,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bau et al., 2017] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, pages 3319-3327, 2017.","title":"Network dissection: Quantifying interpretability of deep visual representations","authors":"David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caelles et al., 2017] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In CVPR, pages 5320-5329, 2017.","title":"One-shot video object segmentation","authors":"Kevis-Kokitsi Sergi Caelles, Jordi Maninis, Laura Pont-Tuset, Daniel Leal-Taixé, Luc Cremers, Van Gool","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018a] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. PAMI, 40(4):834-848, 2018.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Chen","year":"2018","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018b] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for se- mantic image segmentation. In ECCV, pages 801-818, 2018.","title":"Encoder-decoder with atrous separable convolution for semantic image segmentation","authors":"Chen","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K. Su. This looks like that: Deep learning for interpretable image recognition. In NeurIPS, pages 8930-8941, 2019.","title":"This looks like that: Deep learning for interpretable image recognition","authors":"Chen","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Felzenszwalb and Huttenlocher, 2004] Pedro F. Felzen- szwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation. IJCV, 59(2):167-181, 2004.","title":"Efficient graph-based image segmentation","authors":"Pedro F Felzenszwalb, Daniel P Huttenlocher","year":"2004","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghorbani et al., 2019] Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim. Towards automatic concept- based explanations. In NeurIPS, pages 9277-9286, 2019.","title":"Towards automatic conceptbased explanations","authors":"Ghorbani","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv:1803.07728, 2018.","title":"Unsupervised representation learning by predicting image rotations","authors":"Spyros Gidaris, Praveer Singh, Nikos Komodakis","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9726-9735, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang and Li, 2020] Zixuan Huang and Yin Li. Inter- pretable and accurate fine-grained recognition via region grouping. In CVPR, pages 8659-8669, 2020.","title":"Interpretable and accurate fine-grained recognition via region grouping","authors":"Li ; Huang, Zixuan Huang, Yin Li","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hwang et al., 2019] Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chieh Chen. Segsort: Segmentation by discrimi- native sorting of segments. In ICCV, pages 7333-7343, 2019.","title":"Segsort: Segmentation by discriminative sorting of segments","authors":"Hwang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ji et al., 2019] Xu Ji, Andrea Vedaldi, and Joao Henriques. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, pages 9864- 9873, 2019.","title":"Invariant information clustering for unsupervised image classification and segmentation","authors":"Xu Ji, Andrea Vedaldi, Joao Henriques","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ke et al., 2021] Tsung-Wei Ke, Jyh-Jing Hwang, and Stella X. Yu. Universal weakly supervised segmentation by pixel-to-segment contrastive learning. In ICLR, 2021.","title":"Universal weakly supervised segmentation by pixel-to-segment contrastive learning","authors":"Tsung-Wei Ke, Jyh-Jing Hwang, Stella X Yu","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, pages 4186-4195, 2018.","title":"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)","authors":"Kim","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kong and Fowlkes, 2019] Shu Kong and Charless Fowlkes. Multigrid predictive filter flow for unsupervised learning on videos. arXiv:1904.01693, 2019.","title":"Multigrid predictive filter flow for unsupervised learning on videos","authors":"Fowlkes Kong, Shu Kong, Charless Fowlkes","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Noroozi and Favaro, 2016] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pages 69-84, 2016.","title":"Unsupervised learning of visual representations by solving jigsaw puzzles","authors":"Favaro Noroozi, Mehdi Noroozi, Paolo Favaro","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Con- text encoders: Feature learning by inpainting. In CVPR, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Perazzi et al., 2017] Federico Perazzi, Anna Khoreva, Ro- drigo Benenson, Bernt Schiele, and Alexander Sorkine- Hornung. Learning video object segmentation from static images. In CVPR, pages 3491-3500, 2017.","title":"Learning video object segmentation from static images","authors":"Perazzi","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qin et al., 2019] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jager- sand. Basnet: Boundary-aware salient object detection. In CVPR, pages 7471-7481, 2019.","title":"Basnet: Boundary-aware salient object detection","authors":"Qin","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi and Malik, 2000] Jianbo Shi and Jitendra Malik. Nor- malized cuts and image segmentation. PAMI, 22(8):888- 905, 2000.","title":"Normalized cuts and image segmentation","authors":"Jianbo Shi, Jitendra Malik","year":"2000","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tighe and Lazebnik, 2010] Joseph Tighe and Svetlana Lazebnik. Superparsing: Scalable nonparametric image parsing with superpixels. In ECCV, pages 352-365, 2010.","title":"Superparsing: Scalable nonparametric image parsing with superpixels","authors":"Lazebnik Tighe, Joseph Tighe, Svetlana Lazebnik","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[van den Oord et al., 2017] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete repre- sentation learning. In NeurIPS, pages 6309-6318, 2017.","title":"Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Van Gansbeke et al., 2021] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. arXiv:2102.06191, 2021.","title":"Unsupervised semantic segmentation by contrasting object mask proposals","authors":"Van Gansbeke","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vondrick et al., 2018] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In ECCV, pages 402-419, 2018.","title":"Tracking emerges by colorizing videos","authors":"Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, Kevin Murphy","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2019] Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learning correspondence from the cycle-consistency of time. In CVPR, pages 2561-2571, 2019.","title":"Learning correspondence from the cycle-consistency of time","authors":"Xiaolong Wang, Allan Jabri, Alexei A Efros","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Maire, 2020] Xiao Zhang and Michael Maire. Self-supervised visual representation learning from hierar- chical grouping. In NeurIPS, pages 16579-16590, 2020.","title":"Self-supervised visual representation learning from hierarchical grouping","authors":"Xiao Zhang, Michael Maire","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiao- juan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 6230-6239, 2017.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}