{"type":"ScholarlyArticle","title":"SAR-to-Optical Image Translation via Neural Partial Differential Equations","authors":[{"name":"Mingjin Zhang","slug":"mingjin-zhang","given":"Mingjin","family":"Zhang","affiliation":"State Key Laboratory of Integrated Services Networks , School of Telecommunications Engineering , Xidian University , Xi'an 710071 , China; JD Explore Academy , China"},{"name":"Chengyu He","slug":"chengyu-he","given":"Chengyu","family":"He","affiliation":"State Key Laboratory of Integrated Services Networks , School of Telecommunications Engineering , Xidian University , Xi'an 710071 , China"},{"name":"Jing Zhang","slug":"jing-zhang","given":"Jing","family":"Zhang","affiliation":"The University of Sydney , NSW 2006 , Australia"},{"name":"Yuxiang Yang","slug":"yuxiang-yang","given":"Yuxiang","family":"Yang","affiliation":"Hangzhou Dianzi University , Hangzhou 310018 , China"},{"name":"Xiaoqi Peng","slug":"xiaoqi-peng","given":"Xiaoqi","family":"Peng","affiliation":"State Key Laboratory of Integrated Services Networks , School of Telecommunications Engineering , Xidian University , Xi'an 710071 , China"},{"name":"Jie Guo","slug":"jie-guo","given":"Jie","family":"Guo","affiliation":"State Key Laboratory of Integrated Services Networks , School of Telecommunications Engineering , Xidian University , Xi'an 710071 , China"}],"abstract":"Synthetic Aperture Radar (SAR) becomes prevailing in remote sensing while SAR images are challenging to interpret by human visual perception due to the active imaging mechanism and speckle noise. Recent researches on SAR-to-optical image translation provide a promising solution and have attracted increasing attentions, though still suffering from low optical image quality with geometric distortion due to the large domain gap. In this paper, we mitigate this issue from a novel perspective, i.e., neural partial differential equations (PDE). First, based on the efficient numerical scheme for solving PDE, i.e., Taylor Central Difference (TCD), we devise a basic TCD residual block to build the backbone network, which promotes the extraction of useful information in SAR images by aggregating and enhancing features from different levels. Furthermore, inspired by the Perona-Malik Diffusion (PMD), we devise a PMD neural module to implement feature diffusion through layers, aiming at removing the noises in smooth regions while preserving the geometric structures. Assembling them together, we propose a novel SAR-to-Optical image translation network named S2O-NPDE, which delivers optical images with finer structures and less noise while enjoying an explainability advantage from explicit mathematical derivation. Experiments on the popular SEN1-2 dataset show that our model outperforms state-of-the-art methods in terms of both objective metrics and visual quality.","doi":"10.24963/ijcai.2022/229","url":"https://ijcai.org/proceedings/2022/229","pdf":"https://ijcai.org/proceedings/2022/0229.pdf","markdown":"https://ijcai.org/proceedings/2022/229.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/229","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":229,"pages":{"first":1644,"last":1650},"keywords":["Computer Vision: Applications","Computer Vision: Neural generative models, auto encoders, GANs"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986473","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984616","length_sec":354,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anderson and Wendt, 1995] John David Anderson and J Wendt. Computational Fluid Dynamics. Springer, 1995.","title":"Computational Fluid Dynamics","authors":"Anderson, Wendt, John David, Anderson, J Wendt","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2009] Stefan Auer, Stefan Hinz, and Richard Bamler. Ray-tracing simulation techniques for under- standing high-resolution sar images. IEEE Transactions on Geoscience and Remote Sensing, 48(3):1445-1456, 2009.","title":"Ray-tracing simulation techniques for understanding high-resolution sar images","authors":"Stefan Auer, Stefan Hinz, Richard Bamler","year":"2009","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Yunjin Chen, Wei Yu, and Thomas Pock. On learning optimized reaction diffusion processes for ef- fective image restoration. In CVPR, 2015.","title":"On learning optimized reaction diffusion processes for effective image restoration","authors":"Yunjin Chen, Wei Yu, Thomas Pock","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2017] Cong Fang, Zhenyu Zhao, Pan Zhou, and Zhouchen Lin. Feature learning via partial differential equation with applications to face recognition. Pattern Recognition, 69:14-25, 2017.","title":"Feature learning via partial differential equation with applications to face recognition","authors":"Cong Fang, Zhenyu Zhao, Pan Zhou, Zhouchen Lin","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2021] Shilei Fu, Feng Xu, and Ya-Qiu Jin. Recip- rocal translation between sar and optical remote sensing images with cascaded-residual adversarial networks. Sci- ence China Information Sciences, 64(2):1-15, 2021.","title":"Reciprocal translation between sar and optical remote sensing images with cascaded-residual adversarial networks","authors":"Shilei Fu, Feng Xu, Ya-Qiu Jin","year":"2021","venue":"Science China Information Sciences","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fuentes Reyes et al., 2019] Mario Fuentes Reyes, Stefan Auer, Nina Merkle, Corentin Henry, and Michael Schmitt. Sar-to-optical image translation based on conditional gen- erative adversarial networks-optimization, opportunities and limits. Remote Sensing, 11(17):2067, 2019.","title":"Sar-to-optical image translation based on conditional generative adversarial networks-optimization, opportunities and limits","authors":"Mario Fuentes, Reyes, Stefan Auer, Nina Merkle, Corentin Henry, Michael Schmitt","year":"2019","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. Advances in neural information processing systems, 27, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grohnfeldt et al., 2018] Claas Grohnfeldt, Michael Schmitt, and Xiaoxiang Zhu. A conditional generative adversarial network to fuse sar and multispectral optical data for cloud removal from sentinel-2 images. In IGARSS, 2018.","title":"A conditional generative adversarial network to fuse sar and multispectral optical data for cloud removal from sentinel-2 images","authors":"Grohnfeldt","year":"2018","venue":"IGARSS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2011] Zhichang Guo, Jiebao Sun, Dazhi Zhang, and Boying Wu. Adaptive perona-malik model based on the variable exponent for image denoising. IEEE Transac- tions on Image Processing, 21(3):958-967, 2011.","title":"Adaptive perona-malik model based on the variable exponent for image denoising","authors":"Guo","year":"2011","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2021a] Jie Guo, Chengyu He, Mingjin Zhang, Yunsong Li, Xinbo Gao, and Bangyu Song. Edge- preserving convolutional generative adversarial networks for sar-to-optical image translation. Remote Sensing, 13(18):3575, 2021.","title":"Edgepreserving convolutional generative adversarial networks for sar-to-optical image translation","authors":"Guo","year":"2021","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2021b] Ping Guo, Kaizhu Huang, and Zenglin Xu. Partial differential equations is all you need for gen- erating neural architectures-a theory for physical artificial intelligence systems. CoRR abs/2103.08313, 2021.","title":"Partial differential equations is all you need for generating neural architectures-a theory for physical artificial intelligence systems","authors":"Guo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He and Yokoya, 2018] Wei He and Naoto Yokoya. Multi- temporal sentinel-1 and-2 data fusion for optical im- age simulation. ISPRS International Journal of Geo- Information, 7(10):389, 2018.","title":"Multitemporal sentinel-1 and-2 data fusion for optical image simulation","authors":"Yokoya He, Wei He, Naoto Yokoya","year":"2018","venue":"ISPRS International Journal of Geo-Information","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2019] Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, and Jian Cheng. Ode-inspired network design for single image super-resolution. In CVPR, pages 1732-1741, 2019.","title":"Ode-inspired network design for single image super-resolution","authors":"Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, Jian Cheng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Keeling and Stollberger, 2002] Stephen L Keeling and Rudolf Stollberger. Nonlinear anisotropic diffusion filter- ing for multiscale edge enhancement. Inverse Problems, 18(1):175, 2002.","title":"Nonlinear anisotropic diffusion filtering for multiscale edge enhancement","authors":"L Stephen, Rudolf Keeling, Stollberger","year":"2002","venue":"Inverse Problems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2018] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridg- ing deep architectures and numerical differential equa- tions. In ICML, pages 3276-3285, 2018.","title":"Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations","authors":"Yiping Lu, Aoxiao Zhong, Quanzheng Li, Bin Dong","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Perona and Malik, 1990] Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffu- sion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7):629-639, 1990.","title":"Scale-space and edge detection using anisotropic diffusion","authors":"Malik ; Perona, Pietro Perona, Jitendra Malik","year":"1990","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qiao et al., 2019] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image gen- eration by redescription. In CVPR, 2019.","title":"Mirrorgan: Learning text-to-image generation by redescription","authors":"Tingting Qiao, Jing Zhang, Duanqing Xu, Dacheng Tao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schmitt et al., 2018] M Schmitt, LH Hughes, and XX Zhu. The sen1-2 dataset for deep learning in sar-optical data fu- sion. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 4(1), 2018.","title":"The sen1-2 dataset for deep learning in sar-optical data fusion","authors":"Schmitt, Hughes, Zhu","year":"2018","venue":"ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simard et al., 1998] Marc Simard, Gianfranco DeGrandi, Keith PB Thomson, and Goze B Benie. Analysis of speckle noise contribution on wavelet decomposition of sar images. IEEE Transactions on Geoscience and Remote Sensing, 36(6):1953-1962, 1998.","title":"Analysis of speckle noise contribution on wavelet decomposition of sar images","authors":"Marc Simard, Gianfranco Degrandi, Goze B Keith Pb Thomson, Benie","year":"1998","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Patel, 2018] Puyang Wang and Vishal M Patel. Generating high quality visible images from sar images using cnns. In Proceedings of the 2018 IEEE Radar Con- ference, pages 0570-0575, 2018.","title":"Generating high quality visible images from sar images using cnns","authors":"Wang, Patel ; ] Puyang Wang, M Vishal, Patel","year":"2018","venue":"Proceedings of the 2018 IEEE Radar Conference","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans- actions on Image Processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] Lei Wang, Xin Xu, Yue Yu, Rui Yang, Rong Gui, Zhaozhuo Xu, and Fangling Pu. Sar-to-optical image translation using supervised cycle-consistent adver- sarial networks. IEEE Access, 7:129136-129149, 2019.","title":"Sar-to-optical image translation using supervised cycle-consistent adversarial networks","authors":"Lei Wang, Xin Xu, Yue Yu, Rui Yang, Rong Gui, Zhaozhuo Xu, Fangling Pu","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2022] Xi Yang, Jingyi Zhao, Ziyu Wei, Nannan Wang, and Xinbo Gao. Sar-to-optical image translation based on improved cgan. Pattern Recognition, 2022.","title":"Sar-to-optical image translation based on improved cgan","authors":"Xi Yang, Jingyi Zhao, Ziyu Wei, Nannan Wang, Xinbo Gao","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang and Tao, 2020] Jing Zhang and Dacheng Tao. Em- powering things with intelligence: a survey of the progress, challenges, and opportunities in artificial in- telligence of things. IEEE Internet of Things Journal, 8(10):7789-7817, 2020.","title":"Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things","authors":"Tao ; ] Jing Zhang, Dacheng Zhang, Tao","year":"2020","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Jiexin Zhang, Jianjiang Zhou, and Xi- wen Lu. Feature-guided sar-to-optical image translation. IEEE Access, 8:70925-70937, 2020.","title":"Feature-guided sar-to-optical image translation","authors":"Zhang","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In ICCV, pages 2223-2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}