Hybrid Frequency Modulation Network for Image Restoration

1 Introduction

As a fundamental task in computer vision, image restoration involves recovering a high-quality image from its corrupted counterpart by removing degradations and restoring content details [Cui et al., 2023b]. As this task plays an important role in many fields, such as transportation systems, unmanned platforms, and photography, it increasingly garners attention from the industrial community and academia. Due to its ill-posedness property, manifold traditional approaches have been proposed based on various hand-crated features to reduce the solution space. However, these attempts are inapplicable when the assumption is not satisfied.

∗Corresponding author

Recently, methods based on convolutional neural networks (CNNs) have significantly ameliorated the above issue and produced more promising results than conventional approaches. This is achieved by steering clear of human knowledge and learning generalizable priors from large-scale collected datasets. To improve the performance of these networks, many ingenious functional modules have been devised to obtain high-quality predicted images. For example, Qin el al. utilize various attention units, such as channel and pixel attention, for image dehazing, considering totally different weighted information contained in different channel features and uneven haze distribution among the spatial pixels [Qin et al., 2020]. Son et al. leverage multiple atrous convolutions with different rates to deal with spatially-varying defocus blurs [Son et al., 2021]. These advanced mechanisms have significantly boosted the performance of image restoration. Nevertheless, the inherent drawback of convolution, i.e., local connectivity, prohibits its further applications.

Fortunately, inspired by the success of Transformer models in natural language processing and high-level vision tasks, such as object detection and segmentation, many efforts have been made to tailor Transformer for image restoration problems. For instance, Guo et al. first introduce the strengths of Transformer for haze removal [Guo et al., 2022]. To improve the efficiency of Transformer models, some researchers attempt to reduce the operation regions of self-attention [Tsai et al., 2022]. Zamir et al. creatively apply the self-attention operator to the channel dimension rather than spatial regions [Zamir et al., 2022]. Thanks to the powerful ability of self-attention, these models can effectively capture longrange dependencies and have remarkably advanced state-ofthe-art performance for image restoration tasks. Nonetheless, efficiency is a key factor for practical applications, and how to reduce the complexity of self-attention remains a formidable challenge.

To pursue global perceptive fields while remaining highly efficient, a few recent works resort to embedding frequency processing in deep networks. For example, Mao et al. incorporate the fast Fourier transform (FFT) into the residual block to enable both low- and high-frequency learning for image deblurring [Mao et al., 2021]. Guo et al. propose a novel window-based frequency attention to solve the frequency resolution mismatch problem [Guo et al., 2023]. According to the Fourier theorem, these algorithms can effectively model global signals while bridging the frequency gap between degraded and sharp image pairs. However, they only apply the Fourier transform to the spatial dimension, ignoring the importance of channel interactions [Zhang et al., 2023].

To alleviate the above issues, we propose an efficient and effective network based on hybrid dual-dimension frequency modulation. Concretely, to facilitate information exchange between channels that contain different degradation patterns, we develop a novel frequency-based channel feature modulation module by applying FFT to the channel dimension. By doing this, each pixel can perceive signals encompassing different degradation patterns from the same location across all channels. Compared to the pure channel attention that only learns a single attention weight for each channel [Hu et al., 2018], our approach operates at the pixel-wise granularity, offering efficacy in managing spatially varying degradations. Additionally, our module can achieve channel-dimension interactions in multiple spectral spaces by using FFT.

Furthermore, we observe that replacing the direct-current (DC) component of the degraded image with that of the ground truth results in a sharper image, as illustrated in Figure 2. This fact inspires us to propose a frequency-based spatial feature modulation module that only refines the DC component in spectra. To achieve this, we utilize the global average pooling technique to obtain the DC part of the features and then apply lightweight learnable parameters to refine it, which bypasses the use of FFT and IFFT, saving computation overhead. We further inject this mechanism into a densely connected coarse-to-fine paradigm for multi-scale representation learning. Moreover, as the DC component is a kind of low-frequency signal, to facilitate omni-frequency learning, we enhance high-frequency information learning by introducing a frequency-based loss function.

By incorporating the above designs into a U-shaped CNN, the proposed CSNet achieves state-of-the-art performance on several image restoration tasks. For dehazing, our model outperforms the recent Transformer-based method, MB-TaylorFormer-B [Qiu et al., 2023] by 0.63 dB PSNR on the widely-used SOTS [Li et al., 2018] dataset with similar computation overhead, as illustrated in Figure 1. Also, our CSNet shows the strong capability of defocus deblurring by providing a gain of 0.07 dB PSNR over FocalNet [Cui et al., 2023a] in the combined category of the DPDD [Abuolaim and Brown, 2020] dataset. Furthermore, CSNet achieves 38.13 dB PSNR on the CSD [Chen et al., 2021] dataset, an improvement of 0.95 dB over FocalNet [Cui et al., 2023a].

Overall, we summarize the main contributions of this article as follows:

• We propose a frequency-based channel feature modulation module to enhance channel interactions in multiple spectral spaces using the Fourier transformer, enabling each pixel to perceive different degradation patterns from other channels.
• We introduce a multi-scale frequency-based spatial feature modulation module that refines the direct-current component at multiple scales to bring the degraded image closer to ground truth. We also present a frequencybased loss for omni-frequency representation learning.
• Employing hybrid dual-dimension frequency learning, the proposed network achieves state-of-the-art performance on nine datasets for three image restoration tasks.

2 RelatedWork

Image RestorationNetworks. As a fundamental computer vision task, image restoration involves recovering missing details and removing degradations in corrupted images [Cui et al., 2023d]. Traditional approaches mainly stand on various hand-crafted features and assumptions, inevitably facing the issue of inappropriateness in practical scenarios [He et al., 2010]. With the rapid development of deep learning, multifarious CNN-based frameworks have been proposed for diverse image restorations, such as image dehazing [Qin et al., 2020], desnowing [Liu et al., 2018], and deblurring [Son et al., 2021], showcasing more promising performance than the conventional predecessors [Cui et al., 2023c]. Various advanced units and modules have been devised to further boost restoration performance [Cui et al., 2024]. The recent success of Transformer models in high-level vision tasks facilitates the paradigm shift from CNN to Transformer models in image restoration [Qiu et al., 2023; Guo et al., 2022; Song et al., 2022; Valanarasu et al., 2022]. These models have significantly advanced state-of-the-art performance by effectively providing long-distance information interactions.

Attention Mechanisms. Attention mechanisms have been commonly adopted in image restoration tasks to attend to informative regions. For example, Liu et al. adopt the channelwise attention mechanism to flexibly fuse features from different scales for image dehazing [Liu et al., 2019]. Zamir et al. leverage the supervised attention module to control information flow between different stages [Zamir et al., 2021]. Cui et al. introduce a strip attention module to harvest multi-scale contextual information [Cui et al., 2023e]. On the other hand, the recent self-attention module has put the Transformer models in the spotlight [Song et al., 2022; Guo et al., 2022]. Despite a few remedies, the high complexity of self-attention is still an intractable problem. Different from these attention-based methods, we develop lightweight attention modules from the perspective of frequency and adhere to the "channel + spatial" paradigm. Our proposed modules can model global information in spatial and channel dimensions and leverage frequency discrepancies between degraded and clean image pairs.

Spectral Networks. Recently, spectral networks have produced promising results for image restoration by revitalizing frequency processing, which is widely used in traditional algorithms [Mao et al., 2021; Yu et al., 2022; Guo et al., 2023]. The common practice adopted by these methods is first to transfer the spatial features into the spectral domain using the Fourier or Wavelet transforms. The resulting spectra are then refined by a few convolutional layers. The inverse transforms are finally utilized to convent the modulated representation into the spatial domain. The above process is mostly applied in the spatial dimension. In this paper, we present a hybrid dual-dimension frequency learning strategy to enhance channel interactions and refine spatial global information.

3 Method

In this section, we first introduce the overall pipeline of our network. Then, we present the details of the core components: frequency-based channel feature modulation (FCFM), multi-scale frequency-based spatial feature modulation (MFSFM), and frequency-based loss function (FLF).

3.1 Overall Pipeline

As illustrated in Figure 3, our CSNet adopts the widely-used encoder-decoder architecture and consists of three scales for effective multi-scale representation learning. Figure 3 (a) shows that each decoder/encoder contains N normal residual blocks and a modified one that accommodates our MFSFM in the middle of two 3×3 convolutions. Our FCFM is employed in the bottleneck position of CSNet.

Given any input degraded image of shape 3 × H × W, where 3 denotes the number of channels and H ×W specifies the spatial locations in each channel, CSNet first leverages a single 3×3 convolutional layer to produce embedding features of size C ×H ×W, which are then fed into three encoders to obtain the in-depth features. During this process, the channels are expanded, and the resolutions are downsampled using the strided convolutions with the kernel size of 3 and stride of 2. After being refined by our FCFM, the resulting features pass through three decoder blocks to recover sharp features. In this process, the resolution is gradually restored to the original size through transposed convolutions

(kernel size=4, stride=2), while the channel capability is reduced. The residual sharp image is produced via a 3 × 3 convolution, and the final output of CSNet is generated by additionally adding the original degraded input image. Next, we delineate the core components.

3.2 FCFM

Our frequency-based channel feature modulation module (FCFM) is illustrated in Figure 3 (c). It applies FFT among the channel dimensions to enhance channel interactions. As a result, each pixel can integrate information from the same location across all channels in multiple spectral spaces. Furthermore, our module operates at the pixel-wise granularity, which is conducive to managing spatially varying blurs. Specifically, given any input features X ∈ RC×H×W , FFT is used across channels to obtain the real and imaginary components, which are then concatenated among the channel dimension. The concatenated spectra are modulated through a 1 × 1 convolution layer. The output of FCFM is yielded via the channel-dimension IFFT. The above process can be formally expressed as:

where R and I are the real and imaginary components, respectively; [·, ·] denotes the concatenation operation; Conv1×1 represents a convolutional layer with the kernel size of 1 × 1; and C-IFFT applies IFFT among the channel dimension. Although our lightweight FCFM has simple operation, it significantly improves performance over the baseline model, which will be shown in the ablation studies.

3.3 MFSFM

In addition to FCFM, which provides channel-dimension modulation, we further propose a multi-scale frequencybased spatial feature modulation module (MFSFM) to refine spatial features from the frequency perspective. From Figure 2, we can see that replacing the direct-current (DC) component of the degraded image with that of the ground truth leads to a sharper result. This fact inspires us to refine the DC component for spatial feature modulation, which can also model global information. Furthermore, we inject this mechanism, termed FSFM, into a densely connected coarse-to-fine learning paradigm to achieve multi-scale spatial feature refinement. In the following, we first introduce FSFM and then present its multi-scale version.

Similarly, with the input features X ∈ RC×H×W , we utilize the global average pooling to directly obtain the DC component instead of using FFT, resulting in lower computation overhead. The resulting DC part is recalibrated by the lightweight attention parameters optimized by backpropagation. Next, the improved DC part is fused with the remaining features, followed by a depthwise convolution for refinement. The above process can be expressed as:

where GAP is the global average pooling operation; W ∈ RC×1×1 is the channel-wise attention parameters;

DConv3×3 is a 3 × 3 depthwise convolution; and ⊙ denotes the element-wise multiplication, where the size difference between W and features are bridged via the broadcast mechanism of the programming framework.

To enhance multi-scale representation learning, we inject FSFM into a densely connected coarse-to-fine learning paradigm, illustrated in Figure 3 (d). Specifically, the input features are downsampled to multiple feature spaces and then refined by the above FSFM. The resulting features of a branch are delivered to all subsequent branches for feature fusion and coarse-to-fine restoration. The final output of MFSFM is generated by applying a 3 × 3 convolution to the added features from all branches. Similarly, given any input tensor X, the process of MFSFM can be formally expressed as:

where i ∈ {1, 2, 3} indexes the branch; ˆXj represents the upsampled result of a preceding branch; ↑24-i denotes the upsampling operator with the rate of 24-i ; and ˆX0=0.

3.4 FLF

As the DC component can be considered as a kind of lowfrequency signal, we further propose a frequency-based loss function (FLF) as a complementary part of FSFM to refine omni-frequency signals. Denoting the predicted image of CSNet and the ground truth as ˆ I and G, respectively, FLF is given by:

where GAP is the global average pooling technique. By doing this, we bring the high-frequency component of ˆ I closer to that of G. In our case, the high frequency is obtained by removing the DC part from the images.

4 Experiments

In this section, we first introduce the implementation details and evaluation metrics. Then, we compare our results with state-of-the-art algorithms on nine different datasets for three representative image restoration tasks: image dehazing, image defocus deblurring and image desnowing. Ablation experiments are performed in the final part. In the tables, the top-performing scores are highlighted in purple. In the figures, PSNR is computed for comparisons.

4.1 Implementation Details

Unless mentioned otherwise, we adopt the following hyperparameters to train our CSNet. Specifically, the model is trained using the Adam optimizer on 256 × 256 patches with a batch size of 8. The initial learning rate is 2-4 , which is gradually reduced to 1e-6 with cosine annealing. We adopt the horizontal flips for data augmentation. According to the complexity of different tasks, we set N (Figure 3 (b)) to 3 for

33.89 dB FFA-Net

40.74 dB Ours

+∞

GT

Table 3: Nighttime image dehazing comparisons on the GTA5 [Yan et al., 2020] dataset.

dehazing and desnowing, and 15 for deblurring. All experiments are carried out on an NVIDIA Tesla A100 GPU.

We measure the widely-used peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [Wang et al.,

2004] for all datasets, and additionally, adopt the mean absolute error (MAE) and learned perceptual image patch similarity (LPIPS) [Zhang et al., 2018] for the DPDD [Abuolaim and Brown, 2020] dataset.

4.2 Results

Image Dehazing. We evaluate our model on both daytime and nighttime dehazing datasets. For daytime scenes, the numerical results on a synthetic (SOTS [Li et al., 2018]) and three real-world datasets, i.e., Dense-Haze [Ancuti et al., 2019], NH-HAZE [Ancuti et al., 2020], and O-HAZE [Ancuti et al., 2018], are presented in Table 1. As seen, our model produces top-performing results on most metrics across synthetic and real-world datasets. In particular, CSNet outperforms the recent strong Transformer-based algorithm, MBTaylorFormer-B [Qiu et al., 2023], by 0.63 dB PSNR on the SOTS dataset with similar computation overhead, as illustrated in Figure 1. Compared to another recent algorithm, FocalNet [Cui et al., 2023a], our method is more effective in removing real-world haze degradations by providing performance gains of 0.26 dB and 0.10 dB PSNR on Dense-Haze and O-HAZE, respectively. Figure 4 shows the visual results on SOTS [Li et al., 2018]. The image produced by our model is much closer to the ground truth.

In addition, we provide nighttime dehazing results on two datasets, NHR [Zhang et al., 2020] and GTA5 [Yan et al., 2020], in Table 2 and Table 3, respectively. Our model is superior to FocalNet [Cui et al., 2023a] with a performance gain of 0.78 dB in terms of PSNR on NHR. Also, our CSNet outperforms the recent algorithm [Jin et al., 2023] by 1.17 dB PSNR, although it is specially designed for nighttime scenes. Visual comparisons in Figure 5 and Figure 6 illustrate that our model is robust in nighttime scenarios.

Image Defocus Deblurring. For this task, we conduct experiments on the widely used DPDD [Abuolaim and Brown, 2020] dataset. The quantitative results are shown in Table 4. The proposed network outperforms state-of-the-art algorithms in most scenes. Specifically, our model significantly outperforms the strong Transformer-based Restormer [Zamir et al., 2022] by 0.27 dB PSNR in the combined category. In comparison to the recently proposed methods, NRKNet [Quan et al., 2023] and FocalNet [Cui et al., 2023a], our method continues to achieve superior scores, surpassing them by 0.14 dB and 0.07 dB PSNR, respectively. The visual results are shown in Figure 7. We can see that our method recovers more details from the hard blurs than competitors.

Image Desnowing. We evaluate our model on two widely adopted datasets for image desnowing. The numerical scores on the CSD [Chen et al., 2021] and Snow100K [Liu et al., 2018] are presented in Table 5. CSNet generates a 0.18 dB

Table 6: Ablation studies for the proposed components. † denotes models that do not utilize dense connections by eliminating the delivery of features from the first branch to the third branch. It is worth mentioning that our tiny version outperforms the recent Fourmer [Zhou et al., 2023] with lower FLOPs.

PSNR gain over the FocalNet [Cui et al., 2023a] algorithm. On the CSD dataset containing more complex scenes, the advantage is further expanded, suggesting the superiority of our model for snow removal. The visual comparisons on the CSD dataset are illustrated in Figure 8. The result of FocalNet still remains snow blurs. In contrast, our result is much closer to the ground truth and obtains a higher PSNR value.

4.3 Ablation Studies

We experiment to demonstrate the efficacy of the proposed components by training and testing on RESIDE [Li et al., 2018] and SOTS [Li et al., 2018], respectively. The model is trained for 300 epochs with N = 0. Other configurations remain the same as that of our final dehazing model. We obtain the baseline model by removing all proposed components in this tiny CSNet.

Effects of Individual Components. The ablation results for the proposed components are shown in Table 6. The baseline model attains 31.32 dB PSNR on the SOTS [Li et al., 2018] dataset. Our FLF achieves a gain of 0.34 dB PSNR over the baseline model. The channel-dimension processing module, FCFM, significantly improves the performance to 33.42 dB PSNR. Without employing the dense connection, FSFM and MFSFM continue to generate performance improvements, demonstrating the effectiveness of our spatial feature modulation module and multi-scale learning paradigm, respectively. Our complete model obtains the best performance, providing a performance boost of 6.55 dB PSNR over the baseline model. It is worth mentioning that our complete model outperforms the recent Fourmer [Zhou et al., 2023] algorithm with lower complexity.

Comparisons with alternatives to FCFM. We compare our FCFM with popular channel attention mechanisms, such as the squeeze-and-excitation block [Hu et al., 2018] and simplified channel attention [Chen et al., 2022]. The results in Table 7 demonstrate that our method shows superiority to these alternatives by facilitating channel interactions in the spectral domain.

5 Conclusion

In this paper, we present an effective and efficient network, named CSNet, for image restoration based on hybrid frequency modulation, i.e., "channel + spatial" dualdimension representation learning. Specifically, we propose a frequency-based channel feature modulation model to enhance interactions between all channels based on the Fourier transform. Moreover, inspired by our observation, a multiscale frequency-based spatial feature modulation module is developed to refine the direct-current component, which can model the global information and bridge the frequency discrepancies between degraded and sharp image pairs. To achieve omni-frequency learning, a frequency-based loss function is further introduced to train the network. Extensive experiments on nine different benchmark datasets demonstrate that the proposed network achieves state-of-the-art performance for three image restoration tasks.

+∞

GT

Acknowledgements

This work was supported partly by the National Natural Science Foundation of China (Grant No.62322216, 62172409), 2023 CCF-Tencent Rhino-Bird Young Faculty Open Research Fund, and partly by the project "VIDETEC-2" (Grant No.19F2232A) and the Federal Ministry for Digital and Transport of Germany (BMDV).