{"type":"ScholarlyArticle","title":"Cross-Scale Domain Adaptation with Comprehensive Information for Pansharpening","authors":[{"name":"Meiqi Gong","slug":"meiqi-gong","given":"Meiqi","family":"Gong","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Hao Zhang","slug":"hao-zhang","given":"Hao","family":"Zhang","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Hebaixu Wang","slug":"hebaixu-wang","given":"Hebaixu","family":"Wang","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Jun Chen","slug":"jun-chen","given":"Jun","family":"Chen","affiliation":"School of Automation , China University of Geosciences , Wuhan 430074 , China"},{"name":"Jun Huang","slug":"jun-huang","given":"Jun","family":"Huang","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Xin Tian","slug":"xin-tian","given":"Xin","family":"Tian","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Jiayi Ma","slug":"jiayi-ma","given":"Jiayi","family":"Ma","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"}],"abstract":"Deep learning-based pansharpening methods typically use simulated data at the reduced-resolution scale for training. It limits their performance when generalizing the trained model to the full-resolution scale due to incomprehensive information utilization of panchromatic (PAN) images at the full-resolution scale and low generalization ability. In this paper, we adopt two targeted strategies to address the above two problems. On the one hand, we introduce a cross-scale comprehensive information capture module, which improves the information utilization of the original PAN image through fully-supervised reconstruction. On the other hand, we pioneer a domain adaptation strategy to tackle the problem of low generalization across different scales. Considering the instinct domain gap between different scales, we leverage the maximum mean discrepancy loss and the inherent pixel-level correlations between features at different scales to reduce the scale variance, thus boosting the generalization ability of our model. Experiments on various satellites demonstrate the superiority of our method over the state-of-the-arts in terms of information retention. Our code is publicly available at https://github.com/Meiqi-Gong/SDIPS.","doi":"10.24963/ijcai.2024/87","url":"https://ijcai.org/proceedings/2024/87","pdf":"https://ijcai.org/proceedings/2024/0087.pdf","markdown":"https://ijcai.org/proceedings/2024/87.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/87","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":87,"pages":{"first":785,"last":793},"keywords":["Computer Vision: CV: Multimodal learning","Computer Vision: CV: Computational photography"],"videos":[],"references":[{"position":1,"raw":"[Aiazzi et al., 2002] Bruno Aiazzi, Luciano Alparone, Ste- fano Baronti, and Andrea Garzelli. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing, 40(10):2300-2312, 2002.","title":"Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis","authors":"Bruno Aiazzi, Luciano Alparone, Stefano Baronti, Andrea Garzelli","year":"2002","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bonev et al., 2008] Boyan Bonev, Francisco Escolano, and Miguel Cazorla. Feature selection, mutual information, and the classification of high-dimensional patterns: Appli- cations to image classification and microarray data analy- sis. Pattern Analysis and Applications, 11:309-319, 2008.","title":"Feature selection, mutual information, and the classification of high-dimensional patterns: Applications to image classification and microarray data analysis","authors":"Boyan Bonev, Francisco Escolano, Miguel Cazorla","year":"2008","venue":"Pattern Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fu et al., 2019] Xueyang Fu, Zihuang Lin, Yue Huang, and Xinghao Ding. A variational pan-sharpening with local gradient constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10265-10274, 2019.","title":"A variational pan-sharpening with local gradient constraints","authors":"Fu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2017] Yunxing Gao, Hengjian Li, Jiwen Dong, and Guang Feng. A deep convolutional network for medi- cal image super-resolution. In Proceedings of the Chinese Automation Congress, pages 5310-5315, 2017.","title":"A deep convolutional network for medical image super-resolution","authors":"Gao","year":"2017","venue":"Proceedings of the Chinese Automation Congress","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Garzelli et al., 2018] Andrea Garzelli, Bruno Aiazzi, Lu- ciano Alparone, Simone Lolli, and Gemine Vivone. Multispectral pansharpening with radiative transfer-based detail-injection modeling for preserving changes in vege- tation cover. Remote Sensing, 10(8):1308, 2018.","title":"Multispectral pansharpening with radiative transfer-based detail-injection modeling for preserving changes in vegetation cover","authors":"Andrea Garzelli, Bruno Aiazzi, Luciano Alparone, Simone Lolli, Gemine Vivone","year":"2018","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghassemian, 2016] Hassan Ghassemian. A review of re- mote sensing image fusion methods. Information Fusion, 32:75-89, 2016.","title":"A review of remote sensing image fusion methods","authors":"Hassan Ghassemian","year":"2016","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gong et al., 2022] Meiqi Gong, Jiayi Ma, Han Xu, Xin Tian, and Xiao-Ping Zhang. D2tnet: A convlstm network with dual-direction transfer for pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 60:1- 14, 2022.","title":"D2tnet: A convlstm network with dual-direction transfer for pan-sharpening","authors":"Gong","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gong et al., 2023] Meiqi Gong, Hao Zhang, Han Xu, Xin Tian, and Jiayi Ma. Multipatch progressive pansharpening with knowledge distillation. IEEE Transactions on Geo- science and Remote Sensing, 61:1-15, 2023.","title":"Multipatch progressive pansharpening with knowledge distillation","authors":"Gong","year":"2023","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gretton et al., 2012] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.","title":"A kernel two-sample test","authors":"Arthur Gretton, Karsten M Borgwardt, J Malte, Bernhard Rasch, Alexander Schölkopf, Smola","year":"2012","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2015] Yiyong Jiang, Xinghao Ding, Delu Zeng, Yue Huang, and John Paisley. Pan-sharpening with a hyper-laplacian penalty. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pages 540-548, 2015.","title":"Pan-sharpening with a hyper-laplacian penalty","authors":"Jiang","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2022] Cheng Jin, Liang-Jian Deng, Ting-Zhu Huang, and Gemine Vivone. Laplacian pyramid networks: A new approach for multispectral pansharpening. Infor- mation Fusion, 78:158-170, 2022.","title":"Laplacian pyramid networks: A new approach for multispectral pansharpening","authors":"Jin Cheng, Liang-Jian Deng, Ting-Zhu Huang, Gemine Vivone","year":"2022","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2013] Shutao Li, Haitao Yin, and Leyuan Fang. Remote sensing image fusion via sparse representations over learned dictionaries. IEEE Transactions on Geo- science and Remote Sensing, 51(9):4779-4789, 2013.","title":"Remote sensing image fusion via sparse representations over learned dictionaries","authors":"Li","year":"2013","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2020] Jiayi Ma, Wei Yu, Chen Chen, Pengwei Liang, Xiaojie Guo, and Junjun Jiang. Pan-gan: An unsu- pervised pan-sharpening method for remote sensing image fusion. Information Fusion, 62:110-120, 2020.","title":"Pan-gan: An unsupervised pan-sharpening method for remote sensing image fusion","authors":"Jiayi Ma, Wei Yu, Chen Chen, Pengwei Liang, Xiaojie Guo, Junjun Jiang","year":"2020","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Masi et al., 2016] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. Pansharpening by convo- lutional neural networks. Remote Sensing, 8(7):594, 2016.","title":"Pansharpening by convolutional neural networks","authors":"Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, Giuseppe Scarpa","year":"2016","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Restaino et al., 2017] Rocco Restaino, Mauro Dalla Mura, Gemine Vivone, and Jocelyn Chanussot. Context-adaptive pansharpening based on image segmentation. IEEE Trans- actions on Geoscience and Remote Sensing, 55(2):753- 766, 2017.","title":"Context-adaptive pansharpening based on image segmentation","authors":"Rocco Restaino, Mauro Dalla Mura, Gemine Vivone, Jocelyn Chanussot","year":"2017","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tu et al., 2001] Te-Ming Tu, Shun-Chi Su, Hsuen-Chyun Shyu, and Ping S Huang. A new look at ihs-like image fusion methods. Information Fusion, 2(3):177-186, 2001.","title":"A new look at ihs-like image fusion methods","authors":"Te-Ming Tu, Shun-Chi Su, Hsuen-Chyun Shyu, Ping S Huang","year":"2001","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Van Opbroek et al., 2018] Annegreet Van Opbroek, Hakim C Achterberg, Meike W Vernooij, and Marleen De Bruijne. Transfer learning for image segmentation by combining image weighting and kernel learning. IEEE Transactions on Medical Imaging, 38(1):213-224, 2018.","title":"Transfer learning for image segmentation by combining image weighting and kernel learning","authors":"Van Opbroek","year":"2018","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vivone et al., 2014] Gemine Vivone, Luciano Alparone, Jo- celyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A Licciardi, Rocco Restaino, and Lucien Wald. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53(5):2565-2586, 2014.","title":"A critical comparison among pansharpening algorithms","authors":"Vivone","year":"2014","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vivone et al., 2018] Gemine Vivone, Rocco Restaino, and Jocelyn Chanussot. Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Transac- tions on Image Processing, 27(7):3418-3431, 2018.","title":"Gemine Vivone, Rocco Restaino, and Jocelyn Chanussot. Full scale regression-based injection coefficients for panchromatic sharpening","authors":"Vivone","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vivone, 2019] Gemine Vivone. Robust band-dependent spatial-detail approaches for panchromatic sharpening. IEEE Transactions on Geoscience and Remote Sensing, 57(9):6421-6433, 2019.","title":"Gemine Vivone. Robust band-dependent spatial-detail approaches for panchromatic sharpening","authors":"Vivone","year":"2019","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wald et al., 1997] Lucien Wald, Thierry Ranchin, and Marc Mangolini. Fusion of satellite images of different spa- tial resolutions: Assessing the quality of resulting im- ages. Photogrammetric Engineering and Remote Sensing, 63(6):691-699, 1997.","title":"Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images","authors":"Wald","year":"1997","venue":"Photogrammetric Engineering and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiao et al., 2022] Jin-Liang Xiao, Ting-Zhu Huang, Liang- Jian Deng, Zhong-Cheng Wu, and Gemine Vivone. A new context-aware details injection fidelity with adaptive co- efficients estimation for variational pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 60:1- 15, 2022.","title":"A new context-aware details injection fidelity with adaptive coefficients estimation for variational pansharpening","authors":"Jin-Liang Xiao, Ting-Zhu Huang, Liang-Jian Deng, Zhong-Cheng Wu, Gemine Vivone","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and Ma, 2023] Hao Zhang and Jiayi Ma. Stp-som: Scale-transfer learning for pansharpening via estimating spectral observation model. International Journal of Com- puter Vision, 131(12):3226-3251, 2023.","title":"Stp-som: Scale-transfer learning for pansharpening via estimating spectral observation model","authors":"Ma ; ] Hao Zhang, Jiayi Zhang, Ma","year":"2023","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2014] Lefei Zhang, Liangpei Zhang, Dacheng Tao, Xin Huang, and Bo Du. Hyperspectral remote sensing image subpixel target detection based on supervised metric learning. IEEE Transactions on Geoscience and Remote Sensing, 52(8):4955-4965, 2014.","title":"Hyperspectral remote sensing image subpixel target detection based on supervised metric learning","authors":"Zhang","year":"2014","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2021] Hao Zhang, Han Xu, Xin Tian, Junjun Jiang, and Jiayi Ma. Image fusion meets deep learning: A survey and perspective. Information Fusion, 76:323-336, 2021.","title":"Image fusion meets deep learning: A survey and perspective","authors":"Hao Zhang, Han Xu, Xin Tian, Junjun Jiang, Jiayi Ma","year":"2021","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2023] Yuxiang Zhang, Wei Li, Mengmeng Zhang, Ying Qu, Ran Tao, and Hairong Qi. Topolog- ical structure and semantic information transfer network for cross-scene hyperspectral image classification. IEEE Transactions on Neural Networks and Learning Systems, 34(6):2817-2830, 2023.","title":"Topological structure and semantic information transfer network for cross-scene hyperspectral image classification","authors":"Yuxiang Zhang, Wei Li, Mengmeng Zhang, Ying Qu, Ran Tao, Hairong Qi","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2023] Man Zhou, Keyu Yan, Jinshan Pan, Wenqi Ren, Qi Xie, and Xiangyong Cao. Memory- augmented deep unfolding network for guided image super-resolution. International Journal of Computer Vi- sion, 131(1):215-242, 2023.","title":"Memoryaugmented deep unfolding network for guided image super-resolution","authors":"Zhou","year":"2023","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2019] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Jingwu Chen, Zhiping Shi, Wenjuan Wu, and Qing He. Multi-representation adaptation network for cross- domain image classification. Neural Networks, 119:214- 221, 2019.","title":"Multi-representation adaptation network for crossdomain image classification","authors":"Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Jingwu Chen, Zhiping Shi, Wenjuan Wu, Qing He","year":"2019","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2021] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for im- age classification. IEEE Transactions on Neural Networks and Learning Systems, 32(4):1713-1722, 2021.","title":"Deep subdomain adaptation network for image classification","authors":"Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, Qing He","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}