{"type":"ScholarlyArticle","title":"Local-Global Transformer Enhanced Unfolding Network for Pan-sharpening","authors":[{"name":"Mingsong Li","slug":"mingsong-li","given":"Mingsong","family":"Li","affiliation":"School of Software , Shandong University , Jinan , China"},{"name":"Yikun Liu","slug":"yikun-liu","given":"Yikun","family":"Liu","affiliation":"School of Software , Shandong University , Jinan , China"},{"name":"Tao Xiao","slug":"tao-xiao","given":"Tao","family":"Xiao","affiliation":"School of Software , Shandong University , Jinan , China"},{"name":"Yuwen Huang","slug":"yuwen-huang","given":"Yuwen","family":"Huang","affiliation":"School of Computer , Heze University , Heze , China"},{"name":"Gongping Yang","slug":"gongping-yang","given":"Gongping","family":"Yang","affiliation":"School of Software , Shandong University , Jinan , China"}],"abstract":"Pan-sharpening aims to increase the spatial resolution of the low-resolution multispectral (LrMS) image with the guidance of the corresponding panchromatic (PAN) image. Although deep learning (DL)-based pan-sharpening methods have achieved promising performance, most of them have a two-fold deficiency. For one thing, the universally adopted black box principle limits the model interpretability. For another thing, existing DL-based methods fail to efficiently capture local and global dependencies at the same time, inevitably limiting the overall performance. To address these mentioned issues, we first formulate the degradation process of the high-resolution multispectral (HrMS) image as a unified variational optimization problem, and alternately solve its data and prior subproblems by the designed iterative proximal gradient descent (PGD) algorithm. Moreover, we customize a Local-Global Transformer (LGT) to simultaneously model local and global dependencies, and further formulate an LGT-based prior module for image denoising. Besides the prior module, we also design a lightweight data module. Finally, by serially integrating the data and prior modules in each iterative stage, we unfold the iterative algorithm into a stage-wise unfolding network, Local-Global Transformer Enhanced Unfolding Network (LGTEUN), for the interpretable MS pan-sharpening. Comprehensive experimental results on three satellite data sets demonstrate the effectiveness and efficiency of LGTEUN compared with state-of-the-art (SOTA) methods. The source code is available at https://github.com/lms-07/LGTEUN.","doi":"10.24963/ijcai.2023/119","url":"https://ijcai.org/proceedings/2023/119","pdf":"https://ijcai.org/proceedings/2023/0119.pdf","markdown":"https://ijcai.org/proceedings/2023/119.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/119","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":119,"pages":{"first":1071,"last":1079},"keywords":["Computer Vision: CV: Applications","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Aiazzi et al., 2007] Bruno Aiazzi, Stefano Baronti, and Massimo Selva. Improving component substitution pan- sharpening through multivariate regression of ms + pan data. IEEE Transactions on Geoscience and Remote Sens- ing, 45(10):3230-3239, 2007.","title":"Improving component substitution pansharpening through multivariate regression of ms + pan data","authors":"Aiazzi","year":"2007","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ballester et al., 2006] Coloma Ballester, Vicent Caselles, Laura Igual, Joan Verdera, and Bernard Rougé. A varia- tional model for p+ xs image fusion. International Journal of Computer Vision, 69(1):43-58, 2006.","title":"A variational model for p+ xs image fusion","authors":"Coloma Ballester, Vicent Caselles, Laura Igual, Joan Verdera, Bernard Rougé","year":"2006","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for lin- ear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM journal on imaging sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2022] Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool. Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging. Advances in Neural Information Processing Systems, 2022.","title":"Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging","authors":"Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, Luc Van Gool","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chan et al., 2016] Stanley H Chan, Xiran Wang, and Omar A Elgendy. Plug-and-play admm for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3(1):84- 98, 2016.","title":"Plug-and-play admm for image restoration: Fixed-point convergence and applications","authors":"Xiran Stanley H Chan, Omar A Wang, Elgendy","year":"2016","venue":"IEEE Transactions on Computational Imaging","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2022] Zhi-Xuan Chen, Cheng Jin, Tian-Jing Zhang, Xiao Wu, and Liang-Jian Deng. Spanconv: A new convolution via spanning kernel space for lightweight pansharpening. In Proc. 31st Int. Joint Conf. Artif. Intell., pages 1-7, 2022.","title":"Spanconv: A new convolution via spanning kernel space for lightweight pansharpening","authors":"Cheng Zhi-Xuan Chen, Tian-Jing Jin, Xiao Zhang, Liang-Jian Wu, Deng","year":"2022","venue":"Proc. 31st Int. Joint Conf. Artif. Intell","doi":null,"url":null,"internal_url":"/proceedings/2022/118"},{"position":7,"raw":"[Chi et al., 2020] Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. Advances in Neural Information Pro- cessing Systems, 33:4479-4488, 2020.","title":"Fast fourier convolution","authors":"Lu Chi, Borui Jiang, Yadong Mu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chollet, 2017] Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1251-1258, 2017.","title":"Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions","authors":"Chollet","year":"2017","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong et al., 2021] Weisheng Dong, Chen Zhou, Fangfang Wu, Jinjian Wu, Guangming Shi, and Xin Li. Model- guided deep hyperspectral image super-resolution. IEEE Transactions on Image Processing, 30:5754-5768, 2021.","title":"Modelguided deep hyperspectral image super-resolution","authors":"Weisheng Dong, Chen Zhou, Fangfang Wu, Jinjian Wu, Guangming Shi, Xin Li","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations, 2020. Fauvel et al., 2012] Mathieu Fauvel, Yuliya Tarabalka, Jon Atli Benediktsson, Jocelyn Chanussot, and James C Tilton. Advances in spectral-spatial classification of hy- perspectral images. Proceedings of the IEEE, 101(3):652- 675, 2012.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2012","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Frigo and Johnson, 1998] Matteo Frigo and Steven G John- son. Fftw: An adaptive software architecture for the fft. In Proceedings of IEEE International Conference on Acous- tics, Speech and Signal Processing, volume 3, pages 1381- 1384, 1998.","title":"Fftw: An adaptive software architecture for the fft","authors":"Matteo Frigo, Steven G Johnson","year":"1998","venue":"Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fu et al., 2019] Xueyang Fu, Zihuang Lin, Yue Huang, and Xinghao Ding. A variational pan-sharpening with local gradient constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10265-10274, 2019.","title":"A variational pan-sharpening with local gradient constraints","authors":"Fu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hardie et al., 2004] Russell C Hardie, Michael T Eismann, and Gregory L Wilson. Map estimation for hyperspec- tral image resolution enhancement using an auxiliary sen- sor. IEEE Transactions on Image Processing, 13(9):1174- 1184, 2004.","title":"Map estimation for hyperspectral image resolution enhancement using an auxiliary sensor","authors":"Russell C Hardie, Gregory L Michael T Eismann, Wilson","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[King and Wang, 2001] Roger L King and Jianwen Wang. A wavelet based algorithm for pan sharpening landsat 7 im- agery. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, volume 2, pages 849- 851, 2001.","title":"A wavelet based algorithm for pan sharpening landsat 7 imagery","authors":"L Roger, Jianwen King, Wang","year":"2001","venue":"Proceedings of IEEE International Geoscience and Remote Sensing Symposium","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 1833-1844, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu, 2000] JG Liu. Smoothing filter-based intensity mod- ulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing, 21(18):3461-3472, 2000.","title":"Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details","authors":"Liu","year":"2000","venue":"International Journal of Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mou et al., 2022] Chong Mou, Qian Wang, and Jian Zhang. Deep generalized unfolding networks for image restora- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17399- 17410, 2022.","title":"Deep generalized unfolding networks for image restoration","authors":"Chong Mou, Qian Wang, Jian Zhang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Sys- tems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wald et al., 1997] Lucien Wald, Thierry Ranchin, and Marc Mangolini. Fusion of satellite images of different spa- tial resolutions: Assessing the quality of resulting im- ages. Photogrammetric Engineering and Remote Sensing, 63(6):691-699, 1997.","title":"Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images","authors":"Wald","year":"1997","venue":"Photogrammetric Engineering and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2019] Qi Xie, Minghao Zhou, Qian Zhao, Deyu Meng, Wangmeng Zuo, and Zongben Xu. Multispectral and hyperspectral image fusion by ms/hs fusion net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1585-1594, 2019.","title":"Multispectral and hyperspectral image fusion by ms/hs fusion net","authors":"Qi Xie, Minghao Zhou, Qian Zhao, Deyu Meng, Wangmeng Zuo, Zongben Xu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2020] Han Xu, Jiayi Ma, Zhenfeng Shao, Hao Zhang, Junjun Jiang, and Xiaojie Guo. Sdpnet: A deep network for pan-sharpening with enhanced information representation. IEEE Transactions on Geoscience and Re- mote Sensing, 59(5):4120-4134, 2020.","title":"Sdpnet: A deep network for pan-sharpening with enhanced information representation","authors":"Han Xu, Jiayi Ma, Zhenfeng Shao, Hao Zhang, Junjun Jiang, Xiaojie Guo","year":"2020","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2021] Shuang Xu, Jiangshe Zhang, Zixiang Zhao, Kai Sun, Junmin Liu, and Chunxia Zhang. Deep gradient projection networks for pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1366-1375, 2021.","title":"Deep gradient projection networks for pan-sharpening","authors":"Shuang Xu, Jiangshe Zhang, Zixiang Zhao, Kai Sun, Junmin Liu, Chunxia Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2017] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. Pannet: A deep network architecture for pan-sharpening. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 5449-5457, 2017.","title":"Pannet: A deep network architecture for pan-sharpening","authors":"Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, John Paisley","year":"2017","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2022] Gang Yang, Man Zhou, Keyu Yan, Aip- ing Liu, Xueyang Fu, and Fan Wang. Memory-augmented deep conditional unfolding network for pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1788-1797, 2022.","title":"Memory-augmented deep conditional unfolding network for pan-sharpening","authors":"Gang Yang, Man Zhou, Keyu Yan, Aiping Liu, Xueyang Fu, Fan Wang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Ghanem, 2018] Jian Zhang and Bernard Ghanem. Ista-net: Interpretable optimization-inspired deep network for image compressive sensing. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1828-1837, 2018.","title":"Ista-net: Interpretable optimization-inspired deep network for image compressive sensing","authors":"Ghanem ; ] Jian Zhang, Bernard Zhang, Ghanem","year":"2018","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2020] Kai Zhang, Luc Van Gool, and Radu Timofte. Deep unfolding network for image super- resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3217- 3226, 2020.","title":"Deep unfolding network for image superresolution","authors":"Kai Zhang, Luc Van Gool, Radu Timofte","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2022] Xudong Zhao, Mengmeng Zhang, Ran Tao, Wei Li, Wenzhi Liao, Lianfang Tian, and Wilfried Philips. Fractional fourier image transformer for multi- modal remote sensing data classification. IEEE Transac- tions on Neural Networks and Learning Systems, 2022.","title":"Fractional fourier image transformer for multimodal remote sensing data classification","authors":"Xudong Zhao, Mengmeng Zhang, Ran Tao, Wei Li, Wenzhi Liao, Lianfang Tian, Wilfried Philips","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2022a] Huanyu Zhou, Qingjie Liu, and Yun- hong Wang. Panformer: a transformer based model for pan-sharpening. IEEE international conference on multi- media and expo, 2022.","title":"Panformer: a transformer based model for pan-sharpening","authors":"Zhou","year":"2022","venue":"IEEE international conference on multimedia and expo","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2022b] Man Zhou, Jie Huang, Yanchi Fang, Xueyang Fu, and Aiping Liu. Pan-sharpening with cus- tomized transformer and invertible neural network. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 2022.","title":"Pan-sharpening with cus-tomized transformer and invertible neural network","authors":"Zhou","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2022c] Man Zhou, Jie Huang, Keyu Yan, Hu Yu, Xueyang Fu, Aiping Liu, Xian Wei, and Feng Zhao. Spatial-frequency domain information integration for pan-sharpening. In European Conference on Computer Vision, pages 274-291, 2022.","title":"Spatial-frequency domain information integration for pan-sharpening","authors":"Zhou","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2022d] Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual information- driven pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1798-1808, 2022.","title":"Mutual informationdriven pan-sharpening","authors":"Zhou","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2022e] Man Zhou, Hu Yu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy, Deyu Meng, and Chongyi Li. Deep fourier up-sampling. Advances in Neu- ral Information Processing Systems, 2022.","title":"Deep fourier up-sampling","authors":"Zhou","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}