{"type":"ScholarlyArticle","title":"Explore Internal and External Similarity for Single Image Deraining with Graph Neural Networks","authors":[{"name":"Cong Wang","slug":"cong-wang","given":"Cong","family":"Wang","affiliation":"Shenzhen Campus , Sun Yat-sen; The Hong Kong Polytechnic University"},{"name":"Wei Wang","slug":"wei-wang","given":"Wei","family":"Wang","affiliation":"Shenzhen Campus , Sun Yat-sen"},{"name":"Chengjin Yu","slug":"chengjin-yu","given":"Chengjin","family":"Yu","affiliation":"Anhui University; University"},{"name":"Jie Mu","slug":"jie-mu","given":"Jie","family":"Mu","affiliation":"Dongbei University of Finance and Economics"}],"abstract":"Patch-level non-local self-similarity is an important property of natural images. However, most existing methods do not consider this property into neural networks for image deraining, thus affecting recovery performance. Motivated by this property, we find that there exists significant patch recurrence property of a rainy image, that is, similar patches tend to recur many times in one image and its multi-scale images and external images. To better model this property for image detaining, we develop a multi-scale graph network with exemplars, called MSGNN, that contains two branches: 1) internal data-based supervised branch is used to model the internal relations of similar patches from the rainy image itself and its multi-scale images and 2) external data-participated unsupervised branch is used to model the external relations of the similar patches in the rainy image and exemplar. Specifically, we construct a graph model by searching the k-nearest neighboring patches from both the rainy images in a multi-scale framework and the exemplar. After obtaining the corresponding k neighboring patches from the multi-scale images and exemplar, we build a graph and aggregate them in an attentional manner so that the graph can provide more information from similar patches for image deraining. We embed the proposed graph in a deep neural network and train it in an end-to-end manner. Extensive experiments demonstrate that the proposed algorithm performs favorably against eight state-of-the-art methods on five public synthetic datasets and one real-world dataset. The source codes will be available at https://github.com/supersupercong/MSGNN.","doi":"10.24963/ijcai.2024/152","url":"https://ijcai.org/proceedings/2024/152","pdf":"https://ijcai.org/proceedings/2024/0152.pdf","markdown":"https://ijcai.org/proceedings/2024/152.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/152","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":152,"pages":{"first":1371,"last":1379},"keywords":["Computer Vision: CV: Computational photography","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Buades et al., 2005] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. A non-local algorithm for image de- noising. In CVPR, pages 60-65, 2005.","title":"A non-local algorithm for image denoising","authors":"Antoni Buades, Bartomeu Coll, Jean-Michel Morel","year":"2005","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2004] Hong Chang, Dit-Yan Yeung, and Yimin Xiong. Super-resolution through neighbor embed- ding. In CVPR, pages 275-282, 2004.","title":"Super-resolution through neighbor embedding","authors":"Chang","year":"2004","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Hsu, 2013] Yi-Lei Chen and Chiou-Ting Hsu. A generalized low-rank appearance model for spatio- temporally correlated rain streaks. In ICCV, pages 1968- 1975, 2013.","title":"A generalized low-rank appearance model for spatiotemporally correlated rain streaks","authors":"Chen, Hsu ; ] Yi-Lei Chen, Chiou-Ting Hsu","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cui et al., 2022] Xin Cui, Cong Wang, Dongwei Ren, Yun- jin Chen, and Pengfei Zhu. Semi-supervised image deraining using knowledge distillation. IEEE TCSVT, 32(12):8327-8341, 2022.","title":"Semi-supervised image deraining using knowledge distillation","authors":"Xin Cui, Cong Wang, Dongwei Ren, Yunjin Chen, Pengfei Zhu","year":"2022","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dabov et al., 2007] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen O. Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE TIP, 16(8):2080-2095, 2007.","title":"Image denoising by sparse 3-d transform-domain collaborative filtering","authors":"Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, Karen O Egiazarian","year":"2007","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2017] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In CVPR, pages 1715-1723, 2017.","title":"Removing rain from single images via a deep detail network","authors":"Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, John Paisley","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In CVPR, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2020] Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Jun- jun Jiang. Multi-scale progressive fusion network for sin- gle image deraining. In CVPR, pages 8343-8352, 2020.","title":"Multi-scale progressive fusion network for single image deraining","authors":"Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, Junjun Jiang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2022a] Yeying Jin, Wending Yan, Wenhan Yang, and Robby T Tan. Structure representation network and uncertainty feedback learning for dense non-uniform fog removal. In ACCV, pages 2041-2058, 2022.","title":"Structure representation network and uncertainty feedback learning for dense non-uniform fog removal","authors":"Jin","year":"2022","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2022b] Yeying Jin, Wenhan Yang, and Robby T Tan. Unsupervised night image enhancement: When layer decomposition meets light-effects suppression. In ECCV, pages 404-421, 2022.","title":"Unsupervised night image enhancement: When layer decomposition meets light-effects suppression","authors":"Jin","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2023] Yeying Jin, Beibei Lin, Wending Yan, Yuan Yuan, Wei Ye, and Robby T Tan. Enhancing visibility in nighttime haze images using guided apsf and gradient adaptive convolution. In ACM MM, pages 2446-2457, 2023.","title":"Enhancing visibility in nighttime haze images using guided apsf and gradient adaptive convolution","authors":"Yeying Jin, Beibei Lin, Wending Yan, Yuan Yuan, Wei Ye, Robby T Tan","year":"2023","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2016] Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, and Michael S. Brown. Rain streak removal using layer priors. In CVPR, pages 2736-2744, 2016.","title":"Rain streak removal using layer priors","authors":"Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, Michael S Brown","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2018] Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV, pages 262-277, 2018.","title":"Recurrent squeeze-and-excitation context aggregation net for single image deraining","authors":"Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, Hongbin Zha","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2019] Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K. Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, and Xi- aochun Cao. Single image deraining: A comprehensive benchmark analysis. In CVPR, pages 3838-3847, 2019.","title":"Single image deraining: A comprehensive benchmark analysis","authors":"Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K Tokuda, Roberto Hirata, Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, Xiaochun Cao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2015] Yu Luo, Yong Xu, and Hui Ji. Removing rain from a single image via discriminative sparse coding. In ICCV, pages 3397-3405, 2015.","title":"Removing rain from a single image via discriminative sparse coding","authors":"Yu Luo, Yong Xu, Hui Ji","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Plötz and Roth, 2018] Tobias Plötz and Stefan Roth. Neu- ral nearest neighbors networks. In NeurIPS, pages 1095- 1106, 2018.","title":"Neural nearest neighbors networks","authors":"Roth ; Plötz, Tobias Plötz, Stefan Roth","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2019] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image de- raining networks: A better and simpler baseline. In CVPR, pages 3937-3946, 2019.","title":"Progressive image deraining networks: A better and simpler baseline","authors":"Ren","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2004] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assess- ment: from error visibility to structural similarity. IEEE TIP, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019a] Guoqing Wang, Changming Sun, and Arcot Sowmya. Erl-net: Entangled representation learn- ing for single image de-raining. In ICCV, 2019.","title":"Erl-net: Entangled representation learning for single image de-raining","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019b] Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson W. H. Lau. Spa- tial attentive single-image deraining with a high quality real rain dataset. In CVPR, pages 12270-12279, 2019.","title":"Spatial attentive single-image deraining with a high quality real rain dataset","authors":"Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020a] Cong Wang, Yutong Wu, Zhixun Su, and Junyang Chen. Joint self-attention and scale- aggregation for self-calibrated deraining network. In ACM MM, pages 2517-2525, 2020.","title":"Joint self-attention and scaleaggregation for self-calibrated deraining network","authors":"Wang","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020b] Cong Wang, Xiaoying Xing, Yutong Wu, Zhixun Su, and Junyang Chen. DCSFN: deep cross- scale fusion network for single image rain removal. In ACM MM, pages 1643-1651, 2020.","title":"DCSFN: deep crossscale fusion network for single image rain removal","authors":"Wang","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020c] Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. A model-driven deep neural network for sin- gle image rain removal. In CVPR, pages 3100-3109, 2020.","title":"A model-driven deep neural network for single image rain removal","authors":"Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2020d] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca- net: Efficient channel attention for deep convolutional neu- ral networks. In CVPR, pages 11531-11539, 2020.","title":"Ecanet: Efficient channel attention for deep convolutional neural networks","authors":"Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Cong Wang, Xiaoying Xing, Guangle Yao, and Zhixun Su. Single image deraining via deep shared pyramid network. The Visual Computer, 37:1851- 1865, 2021.","title":"Single image deraining via deep shared pyramid network","authors":"Cong Wang, Xiaoying Xing, Guangle Yao, Zhixun Su","year":"2021","venue":"The Visual Computer","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022] Cong Wang, Jinshan Pan, and Xiao- Ming Wu. Online-updated high-order collaborative net- works for single image deraining. In AAAI, volume 36, pages 2406-2413, 2022.","title":"Online-updated high-order collaborative networks for single image deraining","authors":"Cong Wang, Jinshan Pan, Xiao-Ming Wu","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2024a] Cong Wang, Jinshan Pan, Yeying Jin, Liyan Wang, Wei Wang, Gang Fu, Wenqi Ren, and Xi- aochun Cao. How powerful potential of attention on image restoration? arXiv preprint arXiv:2403.10336, 2024.","title":"How powerful potential of attention on image restoration","authors":"Wang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2024b] Cong Wang, Jinshan Pan, Wanyu Lin, Jiangxin Dong, Wei Wang, and Xiao-Ming Wu. Self- promer: Self-prompt dehazing transformers with depth- consistency. In AAAI, volume 38, pages 5327-5335, 2024.","title":"Selfpromer: Self-prompt dehazing transformers with depthconsistency","authors":"Wang","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2024c] Cong Wang, Jinshan Pan, Wei Wang, Jiangxin Dong, Mengzhu Wang, Yakun Ju, and Junyang Chen. Promptrestorer: A prompting image restoration method with degradation perception. NeurIPS, 36, 2024.","title":"Promptrestorer: A prompting image restoration method with degradation perception","authors":"Wang","year":"2024","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2024d] Cong Wang, Jinshan Pan, Wei Wang, Gang Fu, Siyuan Liang, Mengzhu Wang, Xiao-Ming Wu, and Jun Liu. Correlation matching transformation trans- formers for uhd image restoration. In AAAI, volume 38, pages 5336-5344, 2024.","title":"Correlation matching transformation transformers for uhd image restoration","authors":"Wang","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wei et al., 2019] Wei Wei, Deyu Meng, Qian Zhao, Zong- ben Xu, and Ying Wu. Semi-supervised transfer learning for image rain removal. In CVPR, pages 3877-3886, 2019.","title":"Semi-supervised transfer learning for image rain removal","authors":"Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, Ying Wu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu and He, 2022] Kepeng Xu and Gang He. Dnas: A decoupled global neural architecture search method. In CVPR workshops, pages 1979-1985, 2022.","title":"Dnas: A decoupled global neural architecture search method","authors":"He Xu, Kepeng Xu, Gang He","year":"2022","venue":"CVPR workshops","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2019] Kepeng Xu, Yunye Zhang, Wenxin Yu, Zhiqiang Zhang, Jingwei Lu, Yibo Fan, Gang He, and Zhuo Yang. Segmentation of building footprints with xception and iouloss. In ICME workshops, pages 420- 425, 2019.","title":"Segmentation of building footprints with xception and iouloss","authors":"Xu","year":"2019","venue":"ICME workshops","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2023] Kepeng Xu, Gang He, Li Xu, Xingchao Yang, Ming Sun, Yuzhi Wang, Zijia Ma, Haoqiang Fan, and Xing Wen. Towards robust sdrtv-to-hdrtv via dual inverse degradation network. arXiv preprint arXiv:2307.03394, 2023.","title":"Towards robust sdrtv-to-hdrtv via dual inverse degradation network","authors":"Kepeng Xu, Gang He, Li Xu, Xingchao Yang, Ming Sun, Yuzhi Wang, Zijia Ma, Haoqiang Fan, Xing Wen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang et al., 2017] Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single im- age. In CVPR, pages 1685-1694, 2017.","title":"Deep joint rain detection and removal from a single image","authors":"Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, Shuicheng Yan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang et al., 2020] Zongxin Yang, Linchao Zhu, Yu Wu, and Yi Yang. Gated channel transformation for visual recogni- tion. In CVPR, pages 11791-11800, 2020.","title":"Gated channel transformation for visual recognition","authors":"Yang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yasarla et al., 2020] Rajeev Yasarla, Vishwanath A. Sindagi, and Vishal M. Patel. Syn2real transfer learning for image deraining using gaussian processes. In CVPR, pages 2723-2733, 2020.","title":"Syn2real transfer learning for image deraining using gaussian processes","authors":"Rajeev Yasarla, A Vishwanath, Vishal M Sindagi, Patel","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zamir et al., 2021] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progres- sive image restoration. In CVPR, 2021.","title":"Multi-stage progressive image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Ling Yang, Shao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang and Patel, 2018] He Zhang and Vishal M. Patel. Density-aware single image de-raining using a multi- stream dense network. In CVPR, pages 695-704, 2018.","title":"Density-aware single image de-raining using a multistream dense network","authors":"Patel ; ] He Zhang, Vishal M Zhang, Patel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhou et al., 2020] Shangchen Zhou, Jiawei Zhang, Wang- meng Zuo, and Chen Change Loy. Cross-scale inter- nal graph neural network for image super-resolution. In NeurIPS, 2020.","title":"Cross-scale internal graph neural network for image super-resolution","authors":"Zhou","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhou et al., 2021] Man Zhou, Jie Xiao, Yifan Chang, Xueyang Fu, Aiping Liu, Jinshan Pan, and Zheng-Jun Zha. Image de-raining via continual learning. In CVPR, pages 4907-4916, 2021.","title":"Image de-raining via continual learning","authors":"Zhou","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhu et al., 2020] Honghe Zhu, Cong Wang, Yajie Zhang, Zhixun Su, and Guohui Zhao. Physical model guided deep image deraining. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, 2020.","title":"Physical model guided deep image deraining","authors":"Zhu","year":"2020","venue":"IEEE International Conference on Multimedia and Expo (ICME)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}