{"type":"ScholarlyArticle","title":"Region-Aware Metric Learning for Open World Semantic Segmentation via Meta-Channel Aggregation","authors":[{"name":"Hexin Dong","slug":"hexin-dong","given":"Hexin","family":"Dong","affiliation":"Center for Data Science , Peking University , Beijing , China"},{"name":"Zifan Chen","slug":"zifan-chen","given":"Zifan","family":"Chen","affiliation":"Center for Data Science , Peking University , Beijing , China"},{"name":"Mingze Yuan","slug":"mingze-yuan","given":"Mingze","family":"Yuan","affiliation":"Center for Data Science , Peking University , Beijing , China"},{"name":"Yutong Xie","slug":"yutong-xie","given":"Yutong","family":"Xie","affiliation":"Center for Data Science , Peking University , Beijing , China"},{"name":"Jie Zhao","slug":"jie-zhao","given":"Jie","family":"Zhao","affiliation":"Center for Data Science , Peking University , Beijing , China; National Biomedical Imaging Center , Peking University , Beijing , China"},{"name":"Fei Yu","slug":"fei-yu","given":"Fei","family":"Yu","affiliation":"Center for Data Science , Peking University , Beijing , China"},{"name":"Bin Dong","slug":"bin-dong","given":"Bin","family":"Dong","affiliation":"Beijing International Center for Mathematical Research (BICMR) , Peking University , Beijing , China; Center for Machine Learning Research , Peking University , Beijing , China; National Biomedical Imaging Center , Peking University , Beijing , China"},{"name":"Li Zhang","slug":"li-zhang","given":"Li","family":"Zhang","affiliation":"Center for Data Science , Peking University , Beijing , China; National Biomedical Imaging Center , Peking University , Beijing , China"}],"abstract":"As one of the most challenging and practical segmentation tasks, open-world semantic segmentation requires the model to segment the anomaly regions in the images and incrementally learn to segment out-of-distribution (OOD) objects, especially under a few-shot condition. The current state-of-the-art (SOTA) method, Deep Metric Learning Network (DMLNet), relies on pixel-level metric learning, with which the identification of similar regions having different semantics is difficult. Therefore, we propose a method called region-aware metric learning (RAML), which first separates the regions of the images and generates region-aware features for further metric learning. RAML improves the integrity of the segmented anomaly regions. Moreover, we propose a novel meta-channel aggregation (MCA) module to further separate anomaly regions, forming high-quality sub-region candidates and thereby improving the model performance for OOD objects. To evaluate the proposed RAML, we have conducted extensive experiments and ablation studies on Lost And Found and Road Anomaly datasets for anomaly segmentation and the CityScapes dataset for incremental few-shot learning. The results show that the proposed RAML achieves SOTA performance in both stages of open world segmentation. Our code and appendix are available at https://github.com/czifan/RAML.","doi":"10.24963/ijcai.2022/121","url":"https://ijcai.org/proceedings/2022/121","pdf":"https://ijcai.org/proceedings/2022/0121.pdf","markdown":"https://ijcai.org/proceedings/2022/121.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/121","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":121,"pages":{"first":863,"last":869},"keywords":["Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Segmentation","Computer Vision: Representation Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986605","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984748","length_sec":295,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Baur et al., 2018] Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. Deep autoencoding mod- els for unsupervised anomaly segmentation in brain mr im- ages. In MICCAI Brainlesion Workshop, pages 161-169, 2018.","title":"Deep autoencoding models for unsupervised anomaly segmentation in brain mr images","authors":"Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, Nassir Navab","year":"2018","venue":"MICCAI Brainlesion Workshop","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bendale and Boult, 2015] Abhijit Bendale and Terrance Boult. Towards open world recognition. In CVPR, pages 1893-1902, 2015.","title":"Towards open world recognition","authors":"Abhijit Bendale, Terrance Boult","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cen et al., 2021] Jun Cen, Peng Yun, Junhao Cai, Michael Yu Wang, and Ming Liu. Deep metric learning for open world semantic segmentation. In ICCV, pages 15333-15342, 2021.","title":"Deep metric learning for open world semantic segmentation","authors":"Cen","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chan et al., 2021] Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta clas- sification for out-of-distribution detection in semantic seg- mentation. In ICCV, pages 5128-5137, 2021.","title":"Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation","authors":"Chan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discrim- inative reciprocal points. In ECCV, pages 507-522, 2020.","title":"Learning open set network with discriminative reciprocal points","authors":"Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, Yonghong Tian","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gal and Ghahramani, 2016] Yarin Gal and Zoubin Ghahra- mani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050- 1059, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Ghahramani Gal","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hendrycks and Gimpel, 2017] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. In ICLR, 2017.","title":"A baseline for detecting misclassified and out-ofdistribution examples in neural networks","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hendrycks et al., 2019] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Stein- hardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132, 2019.","title":"Scaling out-of-distribution detection for real-world settings","authors":"Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, Dawn Song","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2021] Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic segmenta- tion. In ICCV, pages 16291-16301, 2021.","title":"Region-aware contrastive learning for semantic segmentation","authors":"Hu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Joseph et al., 2021] K J Joseph, Salman Khan, Fahad Shah- baz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In CVPR, 2021.","title":"Towards open world object detection","authors":"K J Joseph, Salman Khan, Fahad Shahbaz Khan, Vineeth N Balasubramanian","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lis et al., 2019] Krzysztof Lis, Krishna Nakka, Pascal Fua, and Mathieu Salzmann. Detecting the unexpected via im- age resynthesis. In ICCV, pages 2152-2161, 2019.","title":"Detecting the unexpected via image resynthesis","authors":"Krzysztof Lis, Krishna Nakka, Pascal Fua, Mathieu Salzmann","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Oreshkin et al., 2018] Boris N Oreshkin, Pau Rodriguez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurIPS, 2018.","title":"Tadam: Task dependent adaptive metric for improved few-shot learning","authors":"Pau Boris N Oreshkin, Alexandre Rodriguez, Lacoste","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pinggera et al., 2016] Peter Pinggera, Sebastian Ramos, Stefan Gehrig, Uwe Franke, Carsten Rother, and Rudolf Mester. Lost and found: detecting small road hazards for self-driving vehicles. In IROS, pages 1099-1106, 2016.","title":"Lost and found: detecting small road hazards for self-driving vehicles","authors":"Peter Pinggera, Sebastian Ramos, Stefan Gehrig, Uwe Franke, Carsten Rother, Rudolf Mester","year":"2016","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Saito et al., 2021] Kuniaki Saito, Ping Hu, Trevor Darrell, and Kate Saenko. Learning to detect every thing in an open world. arXiv preprint arXiv:2112.01698, 2021.","title":"Learning to detect every thing in an open world","authors":"Saito","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.","title":"Facenet: A unified embedding for face recognition and clustering","authors":"Schroff","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sun et al., 2020] Y. Sun, C. Cheng, Y. Zhang, C. Zhang, L. Zheng, Z. Wang, and Y. Wei. Circle loss: A unified per- spective of pair similarity optimization. In CVPR, 2020.","title":"Circle loss: A unified perspective of pair similarity optimization","authors":"Y Sun, C Cheng, Y Zhang, C Zhang, L Zheng, Z Wang, Y Wei","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2021] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Ex- ploring cross-image pixel contrast for semantic segmenta- tion. In ICCV, 2021.","title":"Exploring cross-image pixel contrast for semantic segmentation","authors":"Wang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wen et al., 2016] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning ap- proach for deep face recognition. In ECCV, pages 499- 515, 2016.","title":"A discriminative feature learning approach for deep face recognition","authors":"Yandong Wen, Kaipeng Zhang, Zhifeng Li, Yu Qiao","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xia et al., 2020] Yingda Xia, Yi Zhang, Fengze Liu, Wei Shen, and Alan L Yuille. Synthesize then compare: De- tecting failures and anomalies for semantic segmentation. In ECCV, pages 145-161, 2020.","title":"Synthesize then compare: Detecting failures and anomalies for semantic segmentation","authors":"Yingda Xia, Yi Zhang, Fengze Liu, Wei Shen, Alan L Yuille","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xian et al., 2019] Y. Xian, S. Choudhury, Y. He, B. Schiele, and Z. Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, 2019.","title":"Semantic projection network for zero-and few-label semantic segmentation","authors":"Y Xian, S Choudhury, Y He, B Schiele, Z Akata","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yu et al., 2020] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driv- ing dataset for heterogeneous multitask learning. In CVPR, pages 2636-2645, 2020.","title":"Bdd100k: A diverse driving dataset for heterogeneous multitask learning","authors":"Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, Trevor Darrell","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yuan et al., 2020] Yuhui Yuan, Xiaokang Chen, Xilin Chen, and Jingdong Wang. Segmentation transformer: Object- contextual representations for semantic segmentation. In ECCV, 2020.","title":"Segmentation transformer: Objectcontextual representations for semantic segmentation","authors":"Yuhui Yuan, Xiaokang Chen, Xilin Chen, Jingdong Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhong et al., 2021] Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Re- viving known knowledge for discovering novel visual cat- egories in an open world. In CVPR, pages 9462-9470, 2021.","title":"Openmix: Reviving known knowledge for discovering novel visual categories in an open world","authors":"Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, Nicu Sebe","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}