Local-Global Transformer Enhanced Unfolding Network for Pan-sharpening

1 Introduction

With the development of remote sensing field, multispectral (MS) image is capable of recording more abundant spectral signatures in spectral domain compared with RGB image, and is widely applied in various fields, e.g., environmental monitoring, precision agriculture, and urban development [Hardie et al., 2004; Fauvel et al., 2012]. However, due to the inherent trade-off between spatial and spectral resolution, it is hard to directly acquire high-resolution multispectral (HrMS) images. Pan-sharpening, a vital yet challenging remote sensing image processing task, aims to produce a HrMS image from the coupled low-resolution multispectral (LrMS) and panchromatic (PAN) images.

∗Corresponding author

Formally, the degradation process of the HrMS image Z ∈ RHW×B is often expressed as [Hardie et al., 2004; Xie et al., 2019; Dong et al., 2021]:

where S ∈ Rhw×HW is a linear operator for the spatial blurring and downsampling, R ∈ RB×1 is the spectral response function of the PAN imaging sensor, and Nx and Ny are the introduced noises during the image acquisition of the LrMS image X ∈ Rhw×B and the PAN image Y ∈ RHW×1 , respectively. Here, H and h (H > h), W and w (W > w), and B represent the spatial height, the spatial weight, and the number of spectral bands of the corresponding image, respectively. In the past few decades, many methods have been developed in light of the degradation process in Eq. (1), which can be roughly divided into two categories, i.e., model-based and deep learning (DL)-based.

Typical model-based methods include component substitution (CS) [Aiazzi et al., 2007], multiresolution analysis (MRA) [Liu, 2000; King and Wang, 2001], and variational optimization (VO) [Ballester et al., 2006; Fu et al., 2019]. These methods rely on prior subjective assumptions in the super-resolving process, and show limited model performance and generalization ability in real scenes. Attracted by the impressive success of DL in various vision tasks [He et al., 2016; Chollet, 2017], many DL-based methods have been developed for pan-sharpening, especially convolutional neural network (CNN)-based methods. Owing to the outstanding feature representation in hierarchical manner, DL-based methods are capable of directly learning strong priors, and achieve competitive performance, e.g., PanNet [Yang et al., 2017] and SDPNet [Xu et al., 2020].

Model Interpretability: Despite the strong feature extraction ability and encouraging performance improvement, the weak model interpretability is a longstanding deficiency for DL-based methods due to the adopted black box principle. To this end, deep unfolding networks (DUNs) combine merits of both model-based and DL-based methods, and reasonably formulate end-to-end DL models tailored to the investigated pan-sharpening problem employing the theoretical designing philosophy. For instance, Xu et al. [Xu et al., 2021] developed the first DUN for pan-sharpening, justifying the generative models for LrMS and PAN images.

Local and Global Dependencies: Although existing DUNs strengthen the model interpretability towards the investigated pan-sharpening problem, their potential has been far from fully explored. Here, for DUNs, we claim that a competitive denoiser of the image denoising step would sufficiently complement the data step in each iterative stage. However, limited by the local receptive field, most popular CNNbased denoisers pay less attention to global dependencies, which are as important as local dependencies. Furthermore, global transformer, e.g., Vision Transformer (ViT) [Dosovitskiy et al., 2020], can capture global dependencies, obtaining outstanding performance in vision tasks. Yet, global transformer has nontrivial quadratic computational complexity to input image size due to the computation of global selfattention, which inescapably decreases model efficiency.

Similarly, MDCUN [Yang et al., 2022] employed nonlocal prior and non-local block [Wang et al., 2018] for modeling long-range dependencies, thus showing high computational cost. Besides DUNs, Zhou et al. [Zhou et al., 2022a] proposed a modality-specific PanFormer based on Swin Transformer [Liu et al., 2021]. To capture local and long-range dependencies, Zhou et al. [Zhou et al., 2022b] designed a CNN and transformer dual-branch model, CTINN. However, no matter the serial model, e.g., PanFormer, or the dual-branch model, e.g., CTINN, they both fail to model local and global dependencies in the same layer, which inevitably generates few limitations to the image denoiser or the total pan-sharpening model.

Following the above analysis, in this paper, we develop a transformer-based deep unfolding network, Local-Global Transformer Enhanced Unfolding Network (LGTEUN), for the interpretable MS pan-sharpening. To be specific, we first formulate a unified variational optimization problem in light of the degenerating observation of pan-sharpening, and design an iterative proximal gradient descent (PGD) algorithm to alternately solve its data and prior subproblems. Second, we elaborate a Local-Global Transformer (LGT) as a prior module for image denoising. The key component in each LGT basic block is its token mixer, the Local-Global Mixer (LG Mixer), which consists of a local branch and a global branch. The local branch calculates local window based self-attention in spatial domain, while the global branch extracts global contextual feature representation in frequency domain. Therefore, the LGT-based prior module can simultaneously capture local and global dependencies, and we also design a lightweight data module. Finally, when unfolding the iterative algorithm into the stage-wise unfolding network, LGTEUN, we serially integrate the lightweight data module and the powerful prior module in each iterative stage. Extensive experimental results on three satellite data sets demonstrate the superiority of our method compared with other state-of-the-art (SOTA) methods (as shown in Fig. 1). Our contributions can be summarized as follows:

1. We customize a transformer module LGT as an image denoiser to efficiently model local and global dependencies at the same time and sufficiently mine the potential of the proposed unfolding pan-sharpening framework.
2. We develop an interpretable transformer-based deep unfolding network, LGTEUN. To the best of our knowledge, LGTEUN is the first transformer-based deep unfolding network for the MS pan-sharpening, and LGT is also the first transformer module to perform spatial and frequency dualdomain learning.

2 RelatedWork

In this section, the related deep unfolding networks and transformer-based methods are briefly reviewed.

2.1 Deep Unfolding Network

Through integrating merits of both model-based and DLbased methods, deep unfolding networks (DUNs) much improve the interpretability of DL-based models. DUN unfolds the iterative algorithm tailored to the investigated problem, and optimizes the algorithm employing neural modules in an end-to-end trainable manner. DUN has been utilized to solve different low-level vision tasks, including image denoising [Mou et al., 2022], image compressive sensing [Zhang and Ghanem, 2018], image reconstruction [Cai et al., 2022], and image super-resolution [Xie et al., 2019; Zhang et al., 2020; Dong et al., 2021]. For the discussed MS pan-sharpening, GPPNN [Xu et al., 2021] and MDCUN [Yang et al., 2022] are two representative DUNs. However, restricted by the local receptive field, the adopted CNN-based denoiser in GPPNN pays less attention to global dependencies, which is adverse for reducing copy artifacts. Although MDCUN introduces non-local prior and non-local block to model long-range dependencies, the additional computational cost is heavy. Thus, it is still a crucial issue for DUN to formulate a competitive denoiser of the image denoising step to efficiently capture local and global dependencies and further sufficiently complement the data step in each iterative stage.

2.2 Transformer

Originating from language tasks [Vaswani et al., 2017], transformer has an excellent ability to capture global dependencies, and has been widely applied in various vision tasks, e.g., image classification, object detection, semantic segmentation, and image restoration [Dosovitskiy et al., 2020; Liu et al., 2021; Liang et al., 2021; Zhou et al., 2022a; Zhou et al., 2022b]. However, transformer encounters two main issues. 1) Global transformer has nontrivial quadratic computation complexity to input image size due to the computation of image-level self-attention. 2) Although equipped with considerable-size windows and non-local interactions across windows, local transformer still has difficulties to model image-level global dependencies. Moreover, for pansharpening task, transformer-involved models, e.g., [Zhou et al., 2022a] and [Zhou et al., 2022b], fail to process both local and global dependencies at the same time, which inevitably limits the overall performance.

3 Method

3.1 Model Formulation and Optimization

Technically, under the maximizing a posterior (MAP) framework, recovering the original HrMS image based on the degradation process in Eq. (1) is a typical ill-posed problem. Generally, the estimation of the HrMS image Z is implemented by minimizing the following energy function as

where 12 ∥ X-SZ ∥22 and 12 ∥ Y-ZR ∥22 are the two data fidelity terms coinciding with the degenerating observation, J(Z) is the prior term to constraint the solution space, and λ is a trade-off parameter.

Subsequently, proximal gradient descent (PGD) algorithm [Beck and Teboulle, 2009] is employed to solve Eq. (2) as an iterative convergence problem, i.e.,

where ¯Zk denotes the output of the k-th iteration, and η is the step size. Here, the data terms oriented differentiable operator ∇f(¯Zk-1) is further calculated as

Moreover, this iterative problem can be addressed by alternately solving its data subproblem at the gradient descent step

(Eq. (5)) and its prior subproblem at the proximal mapping step (Eq. (6)). In detail,

where proxη,J represents the proximal operator dependent on the prior term J(·). In this way, the PGD algorithm utilizes a few iterations to alternately update ¯Zk- 1 2 and ¯Zk until convergence. In particular, from a Bayesian perspective, the solution of the prior subproblem Eq. (6) corresponds to a Gaussian denoising problem with noise level √λ [Chan et al., 2016; Zhang et al., 2020; Mou et al., 2022; Cai et al., 2022]. In this work, we elaborate a transformer-based denoiser to approximate the proximal operator proxη,J, which prominently facilitates the denoising capability and further sufficiently complements the data step in each iterative stage.

3.2 Deep Unfolding Network

Through unfolding the iterative PGD algorithm, as illustrated in Fig. 2, we develop our Local-Global Transformer Enhanced Unfolding Network (LGTEUN). The LGTEUN is comprised of several stages. Each stage contains a lightweight CNN-based data module D and a powerful transformer-based prior module P, corresponding to the data subproblem at the gradient descent step (Eq. (5)) and the prior subproblem at the proximal mapping step (Eq. (6)) in each iteration, respectively.

Data Module D

To approximate the closed-form solution of the data subproblem at the gradient descent step (Eq. (5)), we design a lightweight CNN-based data module, i.e.,

Specifically, as shown in Fig. 2 (a), the data module of the k-th stage takes the out of the k -1-th stage ¯Zk-1, the LrMS image 1 X, the PAN image Y, and the stage-specific learnable step size ηk-1 as its module inputs. What's more, the matrix S is implemented by two downsampling units, and each unit consists of a downsampling operation and a 3×3 depth convolution (Conv) layer [Chollet, 2017]. Similarly, the transposed matrix ST is performed by two upsampling units. Besides, one point Conv [Chollet, 2017] is utilized as the matrix R to reduce channels from B to 1, and another point Conv is utilized as the matrix RT for the corresponding inverse channel increase.

Prior Module P

Considering the designing of the denoiser at the image denoising step, previous DUNs are mainly based on CNN, presenting limitations in capturing global dependencies. Here, as the first transformer-based image denoiser in the MS pansharpening oriented DUN, we dedicate significant efforts to craft a Local-Global Transformer (LGT) as the key denoising prior module P, i.e.,

1 As an explanatory instance, the dimension of X is HW × B in mathematical formalization like Eq. (1), and H × W × B in programming implementation here.

(a) LGTEUN

Figure 2: Illustration of the proposed LGTEUN. (a) The overall architecture of LGTEUN with K stages and details of the k-th stage. The lightweight CNN-based data module D and the powerful transformer-based prior module P in each stage correspond to the data and prior subproblems in an iteration of the PGD algorithm. (b) Components of an LGT block. (c) The adopted channel mixer. (d) The key LGMixer is comprised of a local branch and a global branch.

Overall Architecture of LGT. As depicted in Eq. (8) and Fig. 2 (a), given the output ¯Zk- 12 ∈ RH×W×B of D as the module input, LGT applies a U-shaped structure mainly constituted by a series of basic LGT Blocks, and outputs ¯Zk ∈ RH×W×B as the module output. Concretely, LGT first uses a patch embedding layer to split the intermediate image ¯Zk- 12 into non-overlapping patch tokens and further produces the embedded feature X0 ∈ RH×W×C . Second, an encoder-bottleneck-decoder structure extracts the discriminative feature representation Xd ∈ RH×W×C fromX0. In particular, the encoder and decoder both contain two LGT blocks and a resizing unit, and the bottleneck has a single LGT block. In each resizing unit, the downsampling or upsampling operation is responsible for resizing spatial resolution, and a point Conv changes the channel dimension accordingly. Finally, a patch unembedding layer is employed to project Xd to ¯Zk. Here, note that the patch size is set as 1, thus the original pixel vectors in ¯Zk- 12 act as the discussed patch tokens for finer local and global token mixing.

In Fig. 2 (b), each LGT block consists of a layer normalization (LN), a Local-Global Mixer (LG Mixer) for mixing the spatial information, a LN, and a channel mixer in order. As illustrated in Fig. 2 (c), the channel mixer is a depthwise Conv [Chollet, 2017] based neural module for efficient channel mixing. Specifically, the LGMixer as the token mixer is the key component in each LGT block, and Fig. 2 (d) depicts the LG Mixer of the first LG Block in the encoder. For convenience, let Xin ∈ RH×W×C represent the input feature map of our LG Mixer, which is further split into two equal parts Xl ∈ RH×W× C2 and Xg ∈ RH×W× C2 along the channel dimension. Then Xl and Xg are assigned to a local branch and a global branch, respectively. The local branch models local dependencies by computing local window based selfattention in spatial domain, while the global branch captures global dependencies by mining global contextual feature representation in frequency domain. By concatenating the output of the local branch Fl ∈ RH×W× C2 and that of the global branch Fg ∈ RH×W× C2 , the local and global dependencies are simultaneously captured in our LG Mixer.

Local Branch. The local branch calculates local window based multi-head self-attention (WMSA) in spatial domain. In detail, as shown in the left path of Fig. 2 (d), Xl is first partitioned into non-overlapping windows, and each window contains M × M patch tokens. Then the window-specific feature map with HW M2 ×M 2 × C2 dimension is obtained by simply reshaping. Subsequently, three feature embeddings Q, K, and V ∈ R HW M2 ×M2× C2 are generated through a point Conv based linear projection. Furthermore, Q, K, and V are channel-wise divided into h heads, i.e., Q = [Q1 , ...,Qh ], K = [K1 , ...,Kh ], and V = [V1 , ...,Vh ]. Each head contains d = C 2h channels, and Fig. 2 (d) only presents the circumstance with h = 1 for simplification. More importantly, the WMSA map is computed as

where Pi ∈ RM2×M2 is the learnable position embedding. At last, for the feature map Fa, we channel-wise concatenate its h heads and spatially merge its HW M2 windows to yield the branch output Fl.

Global Branch. The global branch extracts global contextual feature representation in frequency domain based on the nature of Fourier transformation. To be specific, according to spectral convolution theorem in Fourier theory [Frigo and Johnson, 1998; Chi et al., 2020; Zhao et al., 2022; Zhou et al., 2022c; Zhou et al., 2022e], feature learning in frequency spectral domain has the image-wide receptive field by channel-wise Fourier transformation. Besides, point-wise multiplications in frequency domain correspond to convolutions in spatial domain. These properties provide vital theoretical guidances of our global branch.

Formally, 2D discrete Fourier transform (DFT) first converts Xg from spatial domain to Fourier frequency domain as the complex component F(Xg), i.e.,

where u and v are frequency components. Here, F(Xg) ∈ CH×( W2 +1)× C2 is produced in light of the conjugate symmetry property of 2D DFT for our real input Xg, and C denotes complex domain. Besides, the inverse 2D DFT is accordingly represented as F -1 (·). Then based on the real part R(Xg) and the imaginary part I(Xg) of F(Xg), the amplitude component A(Xg) and the phase component P(Xg) are further expressed as

Furthermore, two independent 1×1 depth Convs are utilized for feature learning in frequency domain, and the inverse 2D DFT F -1 (·) is applied to recompose the feature representations of the amplitude and phase components back to spatial domain. In detail,

where DConv represents the applied 1×1 depth Conv. In fact, to improve module efficiency, the 2D DFT and the inverse 2D DFT are computed by the 2D real fast Fourier transform (rFFT) and the inverse 2D rFFT, which can be implemented by torch.rfft2 and torch.irfft2 in PyTorch programming framework, respectively. The flowchart of our global branch is depicted in the right path of Fig. 2 (d).

4 Experiments

4.1 Data Sets and EvaluationMetrics

For the MS pan-sharpening, an 8-band MS data set acquired by the WorldView-3 sensor 2 and two 4-band MS data sets acquired by WorldView-2 2 and GaoFen-2 sensors are adopted for experimental analysis. Due to the unavailability of ground-truth (GT) images for training, following Wald's protocol [Wald et al., 1997], we employ downsampling operations to produce a reduced-resolution data set for each satellite sensor. Each data set is further split into non-overlapping subsets for training (about 1000 LrMS/PAN/GT image pairs) and testing (about 140 LrMS/PAN/GT image pairs). The spatial sizes of LrMS, PAN, and GT images are 32 × 32, 128×128, and 128×128, respectively. In addition, we only adopt upsampling operations to produce a full-resolution data set with 120 LrMS/PAN/GT image pairs for the WorldView-3 satellite sensor.

2 https://www.l3harris.com/all-capabilities/ high-resolution-satellite-imagery

For image quality assessment (IQA), five popular metrics are applied for the reduced-resolution test, i.e., PSNR, SSIM, Q-index, SAM, and ERGAS, and three common nonreference metrics are employed for the full-resolution test, i.e., Dλ, DS, and QNR. Besides, the inference time, parameters (Params), and floating-point operations (FLOPs) are utilized for model efficiency analysis.

4.2 Implementation Details

Training Setting. The end-to-end training of LGTEUN is supervised by mean absolute error (MAE) loss between the network output ¯ZK and the GT HrMS image. It trains 130 epochs for the 8-band data set, and 1000 epochs for the two 4band data sets. The Adam optimizer with β1 = 0.9 and β2 = 0.999 is employed for model optimization, and the batch size is set as 4. The initial learning rate is 1.5×10-3 , and decays by 0.85 every 100 epochs. All the experiments are conducted in PyTorch framework with a single NVIDIA GeForce GTX 3090 GPU. For clear comparisons, red color highlights the best results while blue color the second-best in the following suitable table results.

Structure Setting. As shown in Fig. 2 (a), ¯Z0 is initialized by directly upsampling the LrMS image X with a scaling factor 4. In LGTEUN, the data module D shares parameters across stages, while the prior module maintains independence. The channel number C is set as 4B for all the data sets. Additionally, all the downsampling or upsampling operation is implemented by the bicubic interpolation.

Besides, in the local branch of each LGT block, the size of each local window M is 8, and the number of heads is 2. Moreover, in Tab. 1, we explore the impact of the number of iterative stages K = 1, 2, 3, 4 from performance and efficiency viewpoints on 8-band WorldView-3 and 4-band WorldView-2 satellite scenes. According to the results in Tab. 1, it is clear that LGTEUN reaches the optimal performance with 2 stages, and the model efficiency gradually decreases as K increases. Hence, the number of stages is chosen as 2 for better performance and efficiency balance.

4.3 Comparison with SOTAMethods

To comprehensively evaluate the effectiveness and efficiency of the proposed method for the MS pan-sharpening, we compare our LGTEUN with three model-based methods, i.e., GSA [Aiazzi et al., 2007], SFIM [Liu, 2000], and Wavelet [King and Wang, 2001], and six SOTA DL-based methods, i.e., PamFormer [Zhou et al., 2022a], CTINN [Zhou et al., 2022a], LightNet [Chen et al., 2022], SFIIN [Zhou et al., 2022c], MutInf [Zhou et al., 2022d], and MDCUN [Yang et al., 2022]. Besides, all the compared methods are implemented in light of the corresponding paper and source code. It is noteworthy that all the six DL-based methods are the most recent algorithms for the MS pan-sharpening.

Quantitative Comparison. Tab. 2 reports the comparison results of all the discussed ten methods on all the three satellite data sets. Specifically, on all the three data sets, the three model-based methods show limited model performances and generalization abilities, and the DL-based methods obtain more competitive results. More importantly, among all the considered methods on all the three data sets, our proposed LGTEUN always achieves the best results in all the five IQA metrics with distinct performance improvements. For instance, our LGTEUN outperforms the secondbest method by 0.2170 dB, 0.3486 dB, and 0.2687 dB in PSNR on WorldView-3, WorldView-2, and GaoFen-2 data sets, respectively, which indicates the superiority of our proposed method.

Qualitative Comparison. Fig. 3 illustrates the qualitative results of a typical sample from the WorldView-2 data set, including the paired output pan-sharpening image and the corresponding MAE residual image of each discussed method. In Fig. 3, compared with the other nine methods, the proposed LGTEUN exhibits a more visually pleasing result with minor spectral and spatial distortions. In particular, the residual image of our method has fewer artifacts than any other method, especially in the zoom-in region. Here, we can reasonably infer that the advanced performance of the LGTEUN benefits from the designed PGD algorithm based stage iterations and the excellent capability of simultaneously capturing local and global dependencies.

Full-resolution Test. To further measure the model performance in the full-resolution scene, we conduct a fullresolution test on the full-resolution WorldView-3 data set. As reported in Tab. 3, the proposed method also obtains competitive results, i.e., second-best results in DS and QNR and the third-best result in Dλ. On the contrary, MDCUN [Yang et al., 2022] exhibits slightly limited results.

Efficiency Comparison. As for efficiency comparison, Tab. 4 presents exhaustive investigations about inference efficiency (the inference time), model complexity (Params), and computational cost (FLOPs) of all the ten methods on WorldView-3 and GaoFen-2 data sets, and Fig. 1 illustrates the unified PSNR-Params-FLOPs comparisons of all the DLbased methods on the WorldView-2 scenario. Specifically, from Tab. 4, the proposed LGTEUN has excellent inference efficiency and promising computational cost. Besides, further considering the outstanding model performance in Fig. 1, our LGTEUN achieves an impressive performance-efficiency balance.

4.4 Analysis and Discussion

Ablation Study. In this subsection, we perform an ablation study towards our elaborated LGT in the prior module P. Specifically, on the WorldView-3 data set, two break-down ablation tests are conducted to explore and validate the corresponding contributions of its key local and global branches.

Local Branch: For one thing, the local branch models local dependencies by computing local window based self-attention in spatial domain. As reported in Tab. 5, the local branch brings obvious performance gains for both reduced-resolution and full-resolution tests. For example, our LGTEUN improves 0.2879 dB in PSNR and 0.0816 in ERGAS for the reduced-resolution test, and 0.0067 in QNR for the full-resolution test, respectively.

Global Branch: For another thing, the global branch captures global dependencies by mining global contextual feature representation in frequency domain. From Tab. 5, the importance of modeling global dependencies is self-evident since there are distinct performance degradations on all the IQA metrics without the global branch, e.g., 0.2446 dB in PSNR and 0.0026 in Q8 for the reduced-resolution test, and 0.0039 in DS for the full-resolution test, respectively.

Stage-wise Visualization. As illustrated in Fig. 4, for our LGTEUN, we visualize the intermediate results of different stages (¯Z0, ¯Z1, and ¯Z2) from a representative sample in the GaoFen-2 satellite data set, including the paired pansharpening and residual images. It is clear that more detailed information is recovered with LGTEUN iterating.

Limitations. In short, two-fold potential limitations of our LGTEUN are as follows: 1) The pan-sharpening results on the full-resolution scene have room for performance boosting. 2) Further enhancements on model efficiency would make our proposed LGTEUN more competitive.

5 Conclusion

In this paper, for the MS pan-sharpening, we develop our LGTEUN by unfolding the designed PGD optimization algorithm into a deep network to improve the model interpretability. In our LGTEUN, to complement the lightweight data module, we customize a LGT module as a powerful prior module for image denoising to simultaneously capture local and global dependencies. To the best of our knowledge, LGTEUN is the first transformer-based DUN for the MS pansharpening, and LGT is also the first transformer module to perform spatial and frequency dual-domain learning. Comprehensive experimental results on three satellite data sets demonstrate the effectiveness and efficiency of our LGTEUN compared with other SOTA methods.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant U1903127 and in part by the Taishan Industrial Experts Programme under Grant tscy20200303.