{"type":"ScholarlyArticle","title":"MMPN: Multi-supervised Mask Protection Network for Pansharpening","authors":[{"name":"Changjie Chen","slug":"changjie-chen","given":"Changjie","family":"Chen","affiliation":"School of Information Management , Jiangxi University of Finance and Economics , Nanchang , China"},{"name":"Yong Yang","slug":"yong-yang","given":"Yong","family":"Yang","affiliation":"School of Computer Science and Technology , Tiangong University , Tianjin , China"},{"name":"Shuying Huang","slug":"shuying-huang","given":"Shuying","family":"Huang","affiliation":"School of Software , Tiangong University , Tianjin , China"},{"name":"Wei Tu","slug":"wei-tu","given":"Wei","family":"Tu","affiliation":"School of Mathematics and Computer Science , Jiangxi Science and Technology Normal University , Nanchang , China"},{"name":"Weiguo Wan","slug":"weiguo-wan","given":"Weiguo","family":"Wan","affiliation":"School of Software and Internet of Things Engineering , Jiangxi University of Finance and Economics , Nanchang , China"},{"name":"Shengna Wei","slug":"shengna-wei","given":"Shengna","family":"Wei","affiliation":"School of Computer Science and Technology , Tiangong University , Tianjin , China"}],"abstract":"Pansharpening is to fuse a panchromatic (PAN) image with a multispectral (MS) image to obtain a high-spatial-resolution multispectral (HRMS) image. The deep learning-based pansharpening methods usually apply the convolution operation to extract features and only consider the similarity of gradient information between PAN and HRMS images, resulting in the problems of edge blur and spectral distortion in the fusion results. To solve this problem, a multi-supervised mask protection network (MMPN) is proposed to prevent spatial information from being damaged and overcome spectral distortion in the learning process. Firstly, by analyzing the relationships between high-resolution images and corresponding degraded images, a mask protection strategy (MPS) for edge protection is designed to guide the recovery of fused images. Then, based on the MPS, an MMPN containing four branches is constructed to generate the fusion and mask protection images. In MMPN, each branch employs a dual-stream multi-scale feature fusion module (DMFFM), which is built to extract and fuse the features of two input images. Finally, different loss terms are defined for the four branches, and combined into a joint loss function to realize network training. Experiments on simulated and real satellite datasets show that our method is superior to state-of-the-art methods both subjectively and objectively.","doi":"10.24963/ijcai.2023/64","url":"https://ijcai.org/proceedings/2023/64","pdf":"https://ijcai.org/proceedings/2023/0064.pdf","markdown":"https://ijcai.org/proceedings/2023/64.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/64","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":64,"pages":{"first":573,"last":580},"keywords":["Computer Vision: CV: Applications","Computer Vision: CV: Machine learning for vision"],"videos":[],"references":[{"position":1,"raw":"[Aiazzi et al., 2002] Bruno Aiazzi, Luciano Alparone, Ste- fano Baronti, and Andrea Garzelli. Context-driven fu- sion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Trans- actions on geoscience and remote sensing, 40(10):2300- 2312, 2002.","title":"Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis","authors":"Bruno Aiazzi, Luciano Alparone, Stefano Baronti, Andrea Garzelli","year":"2002","venue":"IEEE Transactions on geoscience and remote sensing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aiazzi et al., 2007] Bruno Aiazzi, Stefano Baronti, and Massimo Selva. Improving component substitution pan- sharpening through multivariate regression of ms + pan data. IEEE Transactions on Geoscience and Remote Sens- ing, 45(10):3230-3239, 2007.","title":"Improving component substitution pansharpening through multivariate regression of ms + pan data","authors":"Bruno Aiazzi, Stefano Baronti, Massimo Selva","year":"2007","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2022] Liangjian Deng, Gemine Vivone, Mer- cedes E Paoletti, Giuseppe Scarpa, Jiang He, Yongjun Zhang, Jocelyn Chanussot, and Antonio Plaza. Machine learning in pansharpening: A benchmark, from shallow to deep networks. IEEE Geoscience and Remote Sensing Magazine, 10(3):279-315, 2022.","title":"Machine learning in pansharpening: A benchmark, from shallow to deep networks","authors":"Deng","year":"2022","venue":"IEEE Geoscience and Remote Sensing Magazine","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghassemian, 2016] Hassan Ghassemian. A review of re- mote sensing image fusion methods. Information Fusion, 32:75-89, 2016.","title":"A review of remote sensing image fusion methods","authors":"Hassan Ghassemian","year":"2016","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Liu et al., 2020] Xiangyu Liu, Qingjie Liu, and Yunhong Wang. Remote sensing image fusion based on two-stream fusion network. Information Fusion, 55:1-15, 2020.","title":"Remote sensing image fusion based on two-stream fusion network","authors":"Xiangyu Liu, Qingjie Liu, Yunhong Wang","year":"2020","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lu et al., 2021] Hangyuan Lu, Yong Yang, Shuying Huang, Wei Tu, and Weiguo Wan. A unified pansharpening model based on band-adaptive gradient and detail correc- tion. IEEE Transactions on Image Processing, 31:918- 933, 2021.","title":"A unified pansharpening model based on band-adaptive gradient and detail correction","authors":"Hangyuan Lu, Yong Yang, Shuying Huang, Wei Tu, Weiguo Wan","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Meng et al., 2022] Xiangchao Meng, Nan Wang, Feng Shao, and Shutao Li. Vision transformer for pansharpen- ing. IEEE Transactions on Geoscience and Remote Sens- ing, 60:1-11, 2022.","title":"Vision transformer for pansharpening","authors":"Xiangchao Meng, Nan Wang, Feng Shao, Shutao Li","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ozcelik et al., 2020] Furkan Ozcelik, Ugur Alganci, Elif Sertel, and Gozde Unal. Rethinking cnn-based pansharp- ening: Guided colorization of panchromatic images via gans. IEEE Transactions on Geoscience and Remote Sens- ing, 59(4):3486-3501, 2020.","title":"Rethinking cnn-based pansharpening: Guided colorization of panchromatic images via gans","authors":"Furkan Ozcelik, Ugur Alganci, Elif Sertel, Gozde Unal","year":"2020","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Peng et al., 2022] Siran Peng, Liang-Jian Deng, Jin-Fan Hu, , and Yuwei Zhuo. Source-adaptive discriminative kernels based network for remote sensing pansharpening. International Joint Conferences on Artificial Intelligence (IJCAI), pages 1283-1289, 2022.","title":"Source-adaptive discriminative kernels based network for remote sensing pansharpening","authors":"Siran Peng, Liang-Jian Deng, Jin-Fan Hu, Yuwei Zhuo","year":"2022","venue":"International Joint Conferences on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2022/179"},{"position":10,"raw":"[Tu et al., 2001] Teming Tu, Shunchi Su, Hsuenchyun Shyu, and Ping S Huang. A new look at ihs-like image fusion methods. Information fusion, 2(3):177-186, 2001.","title":"A new look at ihs-like image fusion methods","authors":"Tu","year":"2001","venue":"Information fusion","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Vivone et al., 2020] Gemine Vivone, Mauro Dalla Mura, Andrea Garzelli, Rocco Restaino, Giuseppe Scarpa, Mag- nus O Ulfarsson, Luciano Alparone, and Jocelyn Chanus- sot. A new benchmark based on recent advances in mul- tispectral pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods. IEEE Geoscience and Remote Sensing Magazine, 9(1):53-81, 2020.","title":"A new benchmark based on recent advances in multispectral pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods","authors":"Vivone","year":"2020","venue":"IEEE Geoscience and Remote Sensing Magazine","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Wald, 2000] Lucien Wald. Quality of high resolution syn- thesised images: Is there a simple criterion? In Third con- ference\" Fusion of Earth data: merging point measure- ments, raster maps and remotely sensed images\", pages 99-103. SEE/URISCA, 2000.","title":"Quality of high resolution synthesised images: Is there a simple criterion? In Third conference","authors":"Wald ;, Lucien Wald","year":"2000","venue":"Fusion of Earth data: merging point measurements, raster maps and remotely sensed images","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Wei et al., 2017] Yancong Wei, Qiangqiang Yuan, Huan- feng Shen, and Liangpei Zhang. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geoscience and Remote Sensing Letters, 14(10):1795-1799, 2017.","title":"Boosting the accuracy of multispectral image pansharpening by learning a deep residual network","authors":"Wei","year":"2017","venue":"IEEE Geoscience and Remote Sensing Letters","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Xu et al., 2014] Qizhi Xu, Bo Li, Yun Zhang, and Lin Ding. High-fidelity component substitution pansharpening by the fitting of substitution data. IEEE Transactions on Geo- science and Remote Sensing, 52(11):7380-7392, 2014.","title":"High-fidelity component substitution pansharpening by the fitting of substitution data","authors":"Qizhi Xu, Bo Li, Yun Zhang, Lin Ding","year":"2014","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Yang et al., 2020] Yong Yang, Wei Tu, Shuying Huang, and Hangyuan Lu. Pcdrn: Progressive cascade deep residual network for pansharpening. Remote Sensing, 12(4):676, 2020.","title":"Pcdrn: Progressive cascade deep residual network for pansharpening","authors":"Yong Yang, Wei Tu, Shuying Huang, Hangyuan Lu","year":"2020","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yang et al., 2021] Yong Yang, Wei Tu, Shuying Huang, Hangyuan Lu, Weiguo Wan, and Lixin Gan. Dual-stream convolutional neural network with residual information enhancement for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 60:1-16, 2021.","title":"Dual-stream convolutional neural network with residual information enhancement for pansharpening","authors":"Yong Yang, Wei Tu, Shuying Huang, Hangyuan Lu, Weiguo Wan, Lixin Gan","year":"2021","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yuan et al., 2018] Qiangqiang Yuan, Yancong Wei, Xi- angchao Meng, Huanfeng Shen, and Liangpei Zhang. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE Jour- nal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3):978-989, 2018.","title":"A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening","authors":"Yuan","year":"2018","venue":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang et al., 2022] Tianjiang Zhang, Liangjian Deng, Tingzhu Huang, Jocelyn Chanussot, and Gemine Vivone. A triple-double convolutional neural network for panchro- matic sharpening. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"A triple-double convolutional neural network for panchromatic sharpening","authors":"Zhang","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}