{"type":"ScholarlyArticle","title":"Rainy WCity: A Real Rainfall Dataset with Diverse Conditions for Semantic Driving Scene Understanding","authors":[{"name":"Xian Zhong","slug":"xian-zhong","given":"Xian","family":"Zhong","affiliation":"School of Computer Science and Artificial Intelligence , Wuhan University of Technology; School of Electronic Engineering and Computer Science , Peking University"},{"name":"Shidong Tu","slug":"shidong-tu","given":"Shidong","family":"Tu","affiliation":"School of Computer Science and Artificial Intelligence , Wuhan University of Technology"},{"name":"Xianzheng Ma","slug":"xianzheng-ma","given":"Xianzheng","family":"Ma","affiliation":"School of Computer Science , Wuhan University"},{"name":"Kui Jiang","slug":"kui-jiang","given":"Kui","family":"Jiang","affiliation":"School of Computer Science , Wuhan University"},{"name":"Wenxin Huang","slug":"wenxin-huang","given":"Wenxin","family":"Huang","affiliation":"School of Computer Science and Information Engineering , Hubei University"},{"name":"Zheng Wang","slug":"zheng-wang","given":"Zheng","family":"Wang","affiliation":"School of Computer Science , Wuhan University"}],"abstract":"Scene understanding in adverse weather conditions (e.g. rainy and foggy days) has drawn increasing attention, arising some specific benchmarks and algorithms. However, scene segmentation under rainy weather is still challenging and under-explored due to the following limitations on the datasets and methods: 1) Manually synthetic rainy samples with empirically settings and human subjective assumptions; 2) Limited rainy conditions, including the rain patterns, intensity, and degradation factors; 3) Separated training manners for image deraining and semantic segmentation. To break these limitations, we pioneer a real, comprehensive, and well-annotated scene understanding dataset under rainy weather, named Rainy WCity. It covers various rain patterns and their bring-in negative visual effects, covering wiper, droplet, reflection, refraction, shadow, windshield-blurring, etc. In addition, to alleviate dependence on paired training samples, we design an unsupervised contrastive learning network for real image deraining and the final rainy scene semantic segmentation via multi-task joint optimization. A comprehensive comparison analysis is also provided, which shows that scene understanding in rainy weather is a largely open problem. Finally, we summarize our general observations, identify open research challenges, and point out future directions.","doi":"10.24963/ijcai.2022/243","url":"https://ijcai.org/proceedings/2022/243","pdf":"https://ijcai.org/proceedings/2022/0243.pdf","markdown":"https://ijcai.org/proceedings/2022/243.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/243","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":243,"pages":{"first":1743,"last":1749},"keywords":["Computer Vision: Scene analysis and understanding","Computer Vision: Segmentation","Computer Vision: Applications"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986916","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985059","length_sec":274,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alhaija et al., 2018] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars M. Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes. Int. J. Comput. Vis., 126(9):961-972, 2018.","title":"Augmented reality meets computer vision: Efficient data generation for urban driving scenes","authors":"Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars M Mescheder, Andreas Geiger, Carsten Rother","year":"2018","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11618-11628, 2020.","title":"nuscenes: A multimodal dataset for autonomous driving","authors":"Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2019] Wei-Lun Chang, Hui-Po Wang, Wen- Hsiao Peng, and Wei-Chen Chiu. All about structure: Adapting structural information across domains for boost- ing semantic segmentation. In CVPR, pages 1900-1909, 2019.","title":"All about structure: Adapting structural information across domains for boosting semantic segmentation","authors":"Wei-Lun Chang, Hui-Po Wang, Wen-Hsiao Peng, Wei-Chen Chiu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 833-851, 2018.","title":"Encoderdecoder with atrous separable convolution for semantic image segmentation","authors":"Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Se- bastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In CVPR, pages 3213-3223, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2021] Xueyang Fu, Qi Qi, Zheng-Jun Zha, Yurui Zhu, and Xinghao Ding. Rain streak removal via dual graph convolutional network. In AAAI, pages 1352-1360, 2021.","title":"Rain streak removal via dual graph convolutional network","authors":"Xueyang Fu, Qi Qi, Zheng-Jun Zha, Yurui Zhu, Xinghao Ding","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2021] Xiaowei Hu, Lei Zhu, Tianyu Wang, Chi- Wing Fu, and Pheng-Ann Heng. Single-image real-time rain removal based on depth-guided non-local features. IEEE Trans. Image Process., 30:1759-1770, 2021.","title":"Single-image real-time rain removal based on depth-guided non-local features","authors":"Xiaowei Hu, Lei Zhu, Tianyu Wang, Chi-Wing Fu, Pheng-Ann Heng","year":"2021","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2018] Xun Huang, Ming-Yu Liu, Serge Be- longie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, pages 172-189, 2018.","title":"Multimodal unsupervised image-toimage translation","authors":"Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2020] Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, and Ruigang Yang. The apolloscape open dataset for autonomous driving and its application. IEEE Trans. Pattern Anal. Mach. Intell., 42(10):2702-2719, 2020.","title":"The apolloscape open dataset for autonomous driving and its application","authors":"Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, Ruigang Yang","year":"2020","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2020] Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Jun- jun Jiang. Multi-scale progressive fusion network for single image deraining. In CVPR, pages 8343-8352, 2020.","title":"Multi-scale progressive fusion network for single image deraining","authors":"Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, Junjun Jiang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2021] Jiongchao Jin, Arezou Fatemi, Wallace P. Lira, Fenggen Yu, Biao Leng, Rui Ma, Ali Mahdavi-Amiri, and Hao (Richard) Zhang. RaidaR: A rich annotated image dataset of rainy street scenes. arXiv abs/2104.04606, 2021.","title":"RaidaR: A rich annotated image dataset of rainy street scenes","authors":"Jin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K. Tokuda, Roberto Hirata Junior, Roberto Marcondes Cesar Junior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. Single image deraining: A com- prehensive benchmark analysis. In CVPR, pages 3838- 3847, 2019.","title":"Single image deraining: A comprehensive benchmark analysis","authors":"Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K Tokuda, Roberto Hirata, Junior, Roberto Marcondes, Cesar Junior, Jiawan Zhang, Xiaojie Guo, Xiaochun Cao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liao et al., 2021] Liang Liao, Jing Xiao, Zheng Wang, Chia- Wen Lin, and Shin'ichi Satoh. Image inpainting guided by coherence priors of semantics and textures. In CVPR, pages 6539-6548, 2021.","title":"Image inpainting guided by coherence priors of semantics and textures","authors":"Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, Shin'ichi Satoh","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2022] Xianzheng Ma, Zhixiang Wang, Yacheng Zhan, Yinqiang Zheng, Zheng Wang, Dengxin Dai, and Chia-Wen Lin. Both style and fog matter: Cumulative domain adaptation for semantic foggy scene understanding. In CVPR, 2022.","title":"Both style and fog matter: Cumulative domain adaptation for semantic foggy scene understanding","authors":"Xianzheng Ma, Zhixiang Wang, Yacheng Zhan, Yinqiang Zheng, Zheng Wang, Dengxin Dai, Chia-Wen Lin","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Porav et al., 2020] Horia Porav, Valentina-Nicoleta Musat, Tom Bruls, and Paul Newman. Rainy screens: Collecting rainy datasets, indoors. arXiv abs/2003.04742, 2020.","title":"Rainy screens: Collecting rainy datasets, indoors","authors":"Horia Porav, Valentina-Nicoleta Musat, Tom Bruls, Paul Newman","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sakaridis et al., 2021] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. ACDC: the adverse conditions dataset with correspondences for semantic driving scene understanding. In ICCV, pages 10745-10755, 2021.","title":"ACDC: the adverse conditions dataset with correspondences for semantic driving scene understanding","authors":"Christos Sakaridis, Dengxin Dai, Luc Van Gool","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tung et al., 2017] Frederick Tung, Jianhui Chen, Lili Meng, and James J. Little. The raincouver scene parsing bench- mark for self-driving in adverse weather and at night. IEEE Robotics Autom. Lett., 2(4):2188-2193, 2017.","title":"Little. The raincouver scene parsing benchmark for self-driving in adverse weather and at night","authors":"Frederick Tung, Jianhui Chen, Lili Meng, James J","year":"2017","venue":"IEEE Robotics Autom. Lett","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2021] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell., 43(10):3349-3364, 2021.","title":"Deep high-resolution representation learning for visual recognition","authors":"Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, Bin Xiao","year":"2021","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yu et al., 2020] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In CVPR, pages 2633-2642, 2020.","title":"BDD100K: A diverse driving dataset for heterogeneous multitask learning","authors":"Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, Trevor Darrell","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yu et al., 2021] Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang. BiSeNet V2: bilateral network with guided aggregation for real-time semantic segmentation. Int. J. Comput. Vis., 129(11):3051- 3068, 2021.","title":"BiSeNet V2: bilateral network with guided aggregation for real-time semantic segmentation","authors":"Yu","year":"2021","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zamir et al., 2021] Syed Waqas Zamir, Aditya Arora, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, pages 14821-14831, 2021.","title":"Multi-stage progressive image restoration","authors":"Aditya Syed Waqas Zamir, Salman H Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Ling Yang, Shao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 6230-6239, 2017.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2019] Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn D. Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In CVPR, pages 8856- 8865, 2019.","title":"Improving semantic segmentation via video propagation and label relaxation","authors":"Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn D Newsam, Andrew Tao, Bryan Catanzaro","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}