{"type":"ScholarlyArticle","title":"Exploring Fourier Prior for Single Image Rain Removal","authors":[{"name":"Xin Guo","slug":"xin-guo","given":"Xin","family":"Guo","affiliation":"University of Science and Technology of China , China"},{"name":"Xueyang Fu","slug":"xueyang-fu","given":"Xueyang","family":"Fu","affiliation":"University of Science and Technology of China , China"},{"name":"Man Zhou","slug":"man-zhou","given":"Man","family":"Zhou","affiliation":"University of Science and Technology of China , China"},{"name":"Zhen Huang","slug":"zhen-huang","given":"Zhen","family":"Huang","affiliation":"University of Science and Technology of China , China"},{"name":"Jialun Peng","slug":"jialun-peng","given":"Jialun","family":"Peng","affiliation":"University of Science and Technology of China , China"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"University of Science and Technology of China , China"}],"abstract":"Deep convolutional neural networks (CNNs) have become dominant in the task of single image rain removal. Most of current CNN methods, however, suffer from the problem of overfitting on one single synthetic dataset as they neglect the intrinsic prior of the physical properties of rain streaks. To address this issue, we propose a simple but effective prior - Fourier prior to improve the generalization ability of an image rain removal model. The Fourier prior is a kind of property of rainy images. It is based on a key observation of us - replacing the Fourier amplitude of rainy images with that of clean images greatly suppresses the synthetic and real-world rain streaks. This means the amplitude contains most of the rain streak information and the phase keeps the similar structures of the background. So it is natural for single image rain removal to process the amplitude and phase information of the rainy images separately. In this paper, we develop a two-stage model where the first stage restores the amplitude of rainy images to clean rain streaks, and the second stage restores the phase information to refine fine-grained background structures. Extensive experiments on synthetic rainy data demonstrate the power of Fourier prior. Moreover, when trained on synthetic data, a robust generalization ability to real-world images can also be obtained. The code will be publicly available at https://github.com/willinglucky/ExploringFourier-Prior-for-Single-Image-Rain-Removal.","doi":"10.24963/ijcai.2022/131","url":"https://ijcai.org/proceedings/2022/131","pdf":"https://ijcai.org/proceedings/2022/0131.pdf","markdown":"https://ijcai.org/proceedings/2022/131.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/131","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":131,"pages":{"first":935,"last":941},"keywords":["Computer Vision: Computational photography"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986565","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984708","length_sec":358,"provider":"slideslive"}],"references":[{"position":1,"raw":"Test100 Rain100H Rain100L Test2800 Test1200 Averages Methods PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑ DerainNet [Yang et al., 2017] 22.77 0.810 14.92 0.592 27.03 0.884 24.31 0.861 23.38 0.835 22.48 0.796 SEMI [Wei et al., 2019] 22.35 0.788 16.56 0.486 25.03 0.842 24.43 0.782 26.05 0.822 22.88 0.744 DIDMDN [Zhang and Patel, 2018] 22.56 0.818 17.35 0.524 25.23 0.741 28.13 0.867 29.65 0.901 24.58 0.770 UMRL [Yasarla and Patel, 2019] 24.41 0.829 26.01 0.832 29.18 0.923 29.97 0.905 30.55 0.910 28.02 0.880 RESCAN [Li et al., 2018] 25.00 0.835 26.36 0.786 29.80 0.881 31.29 0.904 30.51 0.882 28.59 0.857 PReNet [Ren et al., 2019] 24.81 0.851 26.77 0.858 32.44 0.950 31.75 0.916 31.36 0.911 29.42 0.897 MSPFN [Jiang et al., 2020] 27.50 0.876 28.66 0.860 32.40 0.933 32.82 0.930 32.39 0.916 30.75 0.903 MPRNet [Zamir et al., 2021] 30.27 0.897 30.41 0.890 36.40 0.965 33.64 0.938 32.91 0.916 32.73 0.921 HINet [Chen et al., 2021] 30.29 0.906 30.65 0.894 37.28 0.970 33.91 0.941 33.05 0.919 33.03 0.926 FPNet (Ours) 30.86 0.915 30.89 0.897 37.96 0.972 34.07 0.943 33.08 0.924 33.37 0.930 References","title":"Test100 Rain100H Rain100L Test2800 Test1200 Averages Methods PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑ PSNR↑ SSIM↑","authors":"[ Derainnet, Yang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Liangyu Chen, Xin Lu, Jie Zhang, Xiao- jie Chu, and Chengpeng Chen. Hinet: Half instance nor- malization network for image restoration. In CVPR, pages 182-192, 2021.","title":"Hinet: Half instance normalization network for image restoration","authors":"Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, Chengpeng Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Comaniciu et al., 2003] Dorin Comaniciu, Visvanathan Ramesh, and Peter Meer. Kernel-based object tracking. IEEE Transactions on pattern analysis and machine intelligence, 25(5):564-577, 2003.","title":"Kernel-based object tracking","authors":"Dorin Comaniciu, Ramesh Visvanathan, Peter Meer","year":"2003","venue":"IEEE Transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Frigo and Johnson, 1998] Matteo Frigo and Steven G John- son. Fftw: An adaptive software architecture for the fft. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP'98 (Cat. No. 98CH36181), volume 3, pages 1381-1384. IEEE, 1998.","title":"Fftw: An adaptive software architecture for the fft","authors":"Matteo Frigo, Steven G Johnson","year":"1998","venue":"Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP'98","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2017a] Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6):2944- 2956, 2017.","title":"Clearing the skies: A deep network architecture for single-image rain removal","authors":"Fu","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2017b] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Remov- ing rain from single images via a deep detail network. In CVPR, pages 3855-3863, 2017.","title":"Removing rain from single images via a deep detail network","authors":"Fu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2010] Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12):2341-2353, 2010.","title":"Single image haze removal using dark channel prior","authors":"He","year":"2010","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn","authors":"He","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2020] Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Jun- jun Jiang. Multi-scale progressive fusion network for sin- gle image deraining. In CVPR, pages 8346-8355, 2020.","title":"Multi-scale progressive fusion network for single image deraining","authors":"Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, Junjun Jiang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kang et al., 2011] Li-Wei Kang, Chia-Wen Lin, and Yu- Hsiang Fu. Automatic single-image-based rain streaks re- moval via image decomposition. IEEE transactions on im- age processing, 21(4):1742-1755, 2011.","title":"Automatic single-image-based rain streaks removal via image decomposition","authors":"Li-Wei Kang, Chia-Wen Lin, Yu-Hsiang Fu","year":"2011","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. Rain streak removal using layer priors. In CVPR, pages 2736-2744, 2016.","title":"Rain streak removal using layer priors","authors":"Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, Michael S Brown","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV, pages 254-269, 2018.","title":"Recurrent squeeze-and-excitation context aggregation net for single image deraining","authors":"Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, Hongbin Zha","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo et al., 2015] Yu Luo, Yong Xu, and Hui Ji. Removing rain from a single image via discriminative sparse coding. In ICCV, pages 3397-3405, 2015.","title":"Removing rain from a single image via discriminative sparse coding","authors":"Yu Luo, Yong Xu, Hui Ji","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oppenheim et al., 1979] A Oppenheim, Jae Lim, Gary Kopec, and SC Pohlig. Phase in speech and pictures. In ICASSP'79. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 4, pages 632-637. IEEE, 1979.","title":"Phase in speech and pictures","authors":"Oppenheim","year":"1979","venue":"ICASSP'79. IEEE International Conference on Acoustics, Speech, and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Quan et al., 2021] Ruijie Quan, Xin Yu, Yuanzhi Liang, and Yi Yang. Removing raindrops and rain streaks in one go. In CVPR, pages 9147-9156, 2021.","title":"Removing raindrops and rain streaks in one go","authors":"Quan","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2019] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image de- raining networks: A better and simpler baseline. In CVPR, pages 3937-3946, 2019.","title":"Progressive image deraining networks: A better and simpler baseline","authors":"Ren","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"International Conference on Medical image computing and computerassisted intervention","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2020] Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. A model-driven deep neural network for sin- gle image rain removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3103-3112, 2020.","title":"A model-driven deep neural network for single image rain removal","authors":"Wang","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wei et al., 2019] Wei Wei, Deyu Meng, Qian Zhao, Zong- ben Xu, and Ying Wu. Semi-supervised transfer learning for image rain removal. In CVPR, pages 3877-3886, 2019.","title":"Semi-supervised transfer learning for image rain removal","authors":"Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, Ying Wu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2021] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In CVPR, pages 14383-14392, 2021.","title":"A fourier-based framework for domain generalization","authors":"Xu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang and Soatto, 2020] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmenta- tion. In CVPR, pages 4085-4095, 2020.","title":"Fda: Fourier domain adaptation for semantic segmentation","authors":"Yang, Soatto ;, Yanchao Yang, Stefano Soatto","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2017] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In CVPR, pages 1357-1366, 2017.","title":"Deep joint rain detection and removal from a single image","authors":"Yang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2020] Yanchao Yang, Dong Lao, Ganesh Sun- daramoorthi, and Stefano Soatto. Phase consistent ecolog- ical domain adaptation. In CVPR, pages 9011-9020, 2020.","title":"Phase consistent ecological domain adaptation","authors":"Yang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yasarla and Patel, 2019] Rajeev Yasarla and Vishal M Patel. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR, pages 8405-8414, 2019.","title":"Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining","authors":"Rajeev Yasarla, M Vishal, Patel","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zamir et al., 2021] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progres- sive image restoration. In CVPR, pages 14821-14831, 2021.","title":"Multi-stage progressive image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Ling Yang, Shao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Patel, 2018] He Zhang and Vishal M Patel. Density-aware single image de-raining using a multi- stream dense network. In CVPR, pages 695-704, 2018.","title":"Density-aware single image de-raining using a multistream dense network","authors":"Patel ; ] He Zhang, Zhang, M Vishal, Patel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using a conditional generative adversarial network. IEEE transactions on circuits and systems for video technology, 30(11):3943- 3956, 2019.","title":"Image de-raining using a conditional generative adversarial network","authors":"Zhang","year":"2019","venue":"IEEE transactions on circuits and systems for video technology","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}