Noise2Grad: Extract Image Noise to Denoise

1 Introduction

Removing noise from an image is a preprocessing step in many imaging projects to facilitate visualization and downstream tasks such as image segmentation and detection. A noisy image x can be represented as

where n denotes the measurement noise, y is the clean image to be restored. This inverse problem is challenging because the statistics of n are usually unknown and complex.

∗Contact Author

1.1 RelatedWork

In the past decades, image denoising has been an active research topic in image processing. Existing image denoising methods can be roughly divided into model-based methods and learning-based methods.

Model-based methods. Most classical image denoising algorithms exploit hand-crafted priors [Xu et al., 2018], [Buades et al., 2005], [Meng and De La Torre, 2013], [Zhao et al., 2014] to guide the denoising process. The non-local self-similarity (NSS) prior [Hou et al., 2020] has demonstrated its powerful ability to aid noise removal. The NSS prior reveals the fact that a natural image contains many similar but non-local patches. Some well-known NSS-based methods include BM3D [Dabov et al., 2007] and WNNM [Gu et al., 2014]. Other prominent techniques, such as wavelet coring [Simoncelli and Adelson, 1996], total variation [Selesnick, 2017] and low-rank assumptions [Zhu et al., 2016] have also been utilized to simplify the denoising problem. Though simple and effective, these model-based methods tend to produce over-smoothed results and cannot handle noise that does not meet their priors.

Learning with paired data. Recently, deep learning has achieved unprecedented success in image denoising. CNNs trained with plenty of noisy/clean image pairs have become the dominant method. One of the seminal networks for image denoising is DnCNN [Zhang et al., 2017], which adopts a residual architecture to ease learning. Beyond DnCNN, many versatile network architectures have been developed to achieve better denoising results, including FFDNet [Zhang et al., 2018], DANet [Yue et al., 2020], NLRN [Liu et al., 2018], VDN [Yue et al., 2019] and RIDNet [Anwar and Barnes, 2019]. These supervised CNNs consistently show impressive performance on some predefined datasets (e.g. synthetic datasets). However, in many imaging systems (e.g. medical imaging, SAR imaging), paired noisy/clean images are difficult to collect, which limits the application of these supervised techniques. To mitigate this issue, Lehtinen et al. [Lehtinen et al., 2018] demonstrate that the denoising CNN can be trained with pairs of independent noisy measurements of the same scene. This elegant training strategy is called Noise2Noise (N2N), and it can achieve denoising performance on par with general supervised learning methods. Nevertheless, it is not always feasible to sample two independent noises for the same scene.

Learning without paired data. To relax the requirement for training data, training CNN denoisers without precollected paired data has become a hot topic [Liu et al., 2019]. Inspired by the Noise2Noise method, Noise2Void [Krull et al., 2019] and Self2Self [Quan et al., 2020] further demonstrate that denoising CNNs can be trained with individual noisy images via the blind spot strategy. These methods are self-supervised because they do not rely on external data to provide supervision. Considering its practical value, the blind spot strategy is further improved in [Laine et al., 2019], [Batson and Royer, 2019], [Wu et al., 2020] to achieve better denoising results. Unfortunately, the effectiveness of these self-supervised methods stems from some pre-defined statistical assumptions, for example, the noise n is zero mean and pixel-independent. This means that these methods cannot cope with noise that violates their assumptions, such as spatially correlated noise. Another elegant strategy is to use unpaired noisy and clean images to learn the transformation between the noise domain and the clean domain. To this end, methods to this category usually integrate noise modeling and removal into the same deep learning framework. For instance, generative adversarial network (GAN) [Chen et al., 2018], [Kaneko and Harada, 2020], [Yan et al., 2020] is widely used to synthesize noise samples corresponding to accessible clean images. These generated data then provide supervision for denoising. However, GAN-based methods are easy to suffer from mode collapse, so that the generated noise is usually unrealistic or lacks diversity. These unrealistic and monotonous noises can lead to poor denoising.

1.2 Motivations

Considering that the collection of unpaired data is easy in most applications and unpaired images contain more information than individual noisy images, the topic of this paper is unpaired denoising. To form supervision, an intuitive idea is to extract noise n from the noisy image x and add it to other clean images to obtain pairs of data. To verify the feasibility of this idea, we first conduct some denoising experiments. We synthesize three noise datasets (original) by software (i.e. Gaussian, Speckle and Poisson), and the ground-truth is available. Then, we subtract the ground-truth from these noisy images to obtain noise components, which are superimposed on random clean images to construct new noise datasets (fake). We train denoising U-Nets on these original and fake datasets respectively, and the results are reported in Table 1. As can be seen, the network trained on the "fake" dataset achieves comparable denoising performance with the network trained on the "original" dataset. Even for signal-dependent noise (i.e. Speckle and Poisson), this strategy is still feasible. These experiments show the effectiveness of our noise superposition strategy. Based on these observations, the focus of this paper is shifted to how to extract noise from noisy images when ground-truth is not available.

On the other hand, we notice that the gradient of the noisy image is dominated by noise (see Figure 2), which inspires us that the gradient of the noisy image can be used to guide the noise extraction while avoiding the interference of other background content. However, the noisy image gradient contains object edges in addition to noise, which may contaminate the extracted noise. How to counteract the negative effect of image gradient is a thorny problem.

1.3 Our Contributions

Based on the above analysis, we develop a new training strategy Noise2Grad to train a noise extraction network. To reduce the interference of the background content of the noisy image, we divide the noise extraction into two subtasks: noise removal and noise approximation. First, the noise removal module roughly removes the noise from the noisy input, and the removed noise is input to a noise approximation module. Since most of the background details have been eliminated by the noise removal module, the noise approximation module can easily produce a realistic noise map to match the gradient of the noisy image. The noise map is multiplied by a random binary (-1 or 1) matrix to destroy the residual structural details, and then superimposed on a random clean image to synthesize a new noisy / clean image pair. These synthesized data are fed back to the noise removal module to guide better noise removal, which indirectly leads to subsequent finer noise extraction. After convergence, the two modules cooperate to extract noise components from the noisy input. Since the noise removal module is trained with data synthesized by our method, we directly use it as the final denoising network instead of retraining one. Through experiments on several denoising tasks, we demonstrate that the denoising performance of Noise2Grad is close to that of CNNs trained with pre-collected paired data and significantly outperforms other self-supervised and unpaired denoising methods.

2 Methodology

Given some unpaired noisy images Dnoise = {xri }Ni=1 and clean images Dclean = {yr j }Mj=1 , the major objective of this paper is extract noise from noisy images and use it to synthesize new noisy data. Hereafter, we use the superscript r to represent real data and s to represent the data synthesized by our method. Our noise extraction network is shown in Figure 3. Since noise extraction is constrained by the image gradient, this training scheme is called Noise2Grad (N2G). In effect, the gradient of the noisy image does not promise high-quality noise extraction. For this problem, the noise extraction network is divided into two parts, a noise removal module and a noise approximation module. These two modules use each other's output to correct their own learning, and cooperate to achieve fine noise extraction.

2.1 Noise Extraction

Image gradient reflects the high-frequency content but excludes the low-frequency part of the image. Generally, the gradient of a noisy image is mainly composed of the gradient of noise. This inspires us that the gradient of the noisy image can be a hint for noise extraction. We calculate the image gradient by combining the horizontal and vertical deviations of adjacent pixels,

where ∗ indicates the convolution operation, ∇ represents the image gradient. Then, the noise extraction network aims to minimize the following gradient similarity loss,

where L2 loss is adopted. ˜n is the output of noise approximation module, it can be expressed as,

where xr is the noisy input, n is the noise component of xr that we want, ˜ ε represents some structural details from xr , F(·) and G(·) represent noise removal module and noise approaximation module, respectively.

Following Eq.(3), the gradient of ˜n is always similar to that of the noise component n. However, some structural details may remain in ˜n (i.e. ‖˜ ε‖ ̸= 0 ), because the label ∇xr contains object edges in addition to noise. Our goal is to obtain the noise component n, so we have to remove the residual structural detail ˜ ε in ˜n. Fortunately, this can be achieved by reducing the structural information contained in the input of the noise approximation module. If the input of the noise approximation module is only noise without other structural information of xr (i.e. ˆn = n ), then its output ˜n should be mainly composed of noise, and other structural details are unlikely to be retained. To do this, we apply a simple feedback technique to turn ˆn into a "clean" noise. We add the noise map ˜n to a random clean image yr to synthesis a new noisy image xs ,

xs and xr have similar noise, but their image content is different. More importantly, xs has a clean label yr , so they can be paired to guide denoising. xs is fed back to the noise removal module to obtain a denoised result F(xs ). The corresponding noise removal objective function is

Combining Eqs.(3) and (6), the overall loss function is,

In this setting, the noise removal module learns to restore image details while removing noise. This means that the ˆn removed by the noise removal module is mainly composed of noise i.e.

where ˆ ε denotes some structural details of xr . Since much image background information is excluded, the noise approximation module is easier to produce realistic noise map. With the noise removal module doing better in restoring image details, we want ‖ˆ ε‖ to converge to 0.

2.2 Detail Destruction

At each training step t, a new noisy image xs can be obtained by noise superposition Eq.(5). Since ˜n may contain some structural details ˜ ε from xr , the synthesized noisy image xs may not be realistic. The unrealistic xs may impair the ability of the noise removal module to restore image details and remove noise. To solve this problem, ˜n is multiplied by a random binary mask m before being superimposed on the clean image yr . Eq.(5) can be reformulated as

where ⊙ represents element-wise multiplication, m has the same dimension as ˜n. Each element in m is set to 1 with a probability of 0.5, otherwise is -1. The random mask m can destroy the residual structural information in the ˜n to obtain a more realistic xs (see Figure 4, (l)-(n)). The realistic xs is beneficial for the noise removal module to learn noise removal and image detail restoration. In addition, multiplying with m can be seen as a way of data augmentation. Since m is random, various noises can be obtained.

2.3 Update Delay

As the noise removal module becomes better at restoring image details while removing noise, the structural information ˆ ε remaining in ˆn gradually decays. Since the noise approximation module takes ˆn as the input, the weakening of ˆ ε will also lead to the attenuation of ˜ ε in ˜n (see Figure 4, (b)-(d) and (e)-(g)). However, the gradient similarity loss Lgrad Eq.(3) hinders this convergence process due to the edge information in the the noisy image gradient (i.e. ∇xr ). To alleviate the negative impact of Lgrad, we gradually reduce the frequency of calculating Lgrad, that is, the time interval τ for calculating Lgrad becomes longer. τ can be further expressed as

where t denotes the training step, ⌊·⌋ is the floor function.

By gradually delaying the calculation of Lgrad, the noise removal module can focus on preserving image details while removing noise. After convergence, the ˆn removed by the noise removal module contains little structural details. Due to the lack of sufficient structural information, the noise approximation module can only output a realistic noise map ˜n = G(ˆn) ≈ n.

Note that, we use the noise removal module as our final denoising network.

2.4 Architecture and Training Details

The implementation of N2G is based on CNNs. For simplicity, we adopt a simple U-Net [Ronneberger et al., 2015] as the noise removal module, while the noise approximation module is only composed of one 1 × 1 convolution layer. We use PyTorch and Adam with a batch size of 1 to train the network. The training images are randomly cropped into 128 × 128 patches before being input to the network. The learning rate if fixed to 0.0002 for the first 2, 500, 000 iterations and linearly decays to 0 for the next 2, 500, 000 iterations. 1

1 We will release our code and datasets soon.

3 Experiments

We evaluate the effectiveness of Noise2Grad on several image denoising tasks in this section.

3.1 Synthetic Noises

Our Noise2Grad requires some unpaired noisy and clean images to train the network. We use the 4744 natural images in [Ma et al., 2016] to synthesize noisy images (i.e. Dnoise ) with software. In addition, 5000 clean images collected from the Internet are adopted as the clean set Dclean . We compare N2G with several state-of-the-art denoising methods, including model-based methods BM3D [Dabov et al., 2007] and NLH [Hou et al., 2020], selflearning methods Noise2Void(N2V) [Krull et al., 2019] and Self2Self(S2S) [Quan et al., 2020], an unpaired learning method LIR [Du et al., 2020], other deep learning methods include Noise2Noise(N2N) [Lehtinen et al., 2018] and a common fully-supervised U-Net. For the sake of fairness, N2N, U-Net and our N2G adopt the same network architecture to perform denoising. BSD300 [Martin et al., 2001] is the test set of the following experiments.

Gaussian noise. We first conduct the comparative experiments on additive gaussian noise. We add zero-mean gaussian noise with a random standard deviation σ ∈ (0, 50] to each training example. For the test sets, noisy images are synthesized in two ways: a fixed noise level σ = 25 and a variable σ ∈ (0, 50]. PSNR comparisons are reported in Table 2. Subjective denoising results are shown in Figure 5.

Speckle noise. Multiplicative speckle noise, often observed in medical ultrasonic images and radar images. It is harder to remove than Gaussian noise because it is signal dependent.

This noise can be modeled by random value multiplications with pixel values of the latent singal y and can be expressed as x = y + y · n. In this model, n is an uniform noise with a mean of 0 and a variance of v. We vary v ∈ (0, 0.2] to synthesize training images. Quantitative and visual comparisons are shown in Table 2 and Figure 6, respectively.

Poisson noise. We then consider poisson noise, which can be used to model photon noise in imaging sensors. The expected magnitude of poisson noise is dependent on the pixel brightness. Following the setting in [Laine et al., 2019], we randomize the noise level λ ∈ [5, 50] separately for each training example. We show the comparisons in Table 2.

Discussion. From the above experiments, we see that the denoising performance of N2G is close to supervised methods (U-Net and N2N), and significantly outperforms other denoising methods (e.g. BM3D and N2V). The effectiveness of N2G does not rely on paired data or prior knowledge about the noise. The only prerequisite is some unpaired noisy and clean images, which is easy to achieve in most practical applications. For various types of noise, N2G consistently exhibits satisfactory denoising performance. The denoised images are clean and sharp. Besides, the denoising results of N2G are close to the results in Table 1 (Fake). This indicates that the noise extracted by N2G is similar to the real noise component of the noisy image. These experiments show that N2G is a promising solution for various denoising tasks.

3.2 Ablation Study

Detail destruction and update delay and are two key components of the N2G training scheme. We then explore the impact of these two technologies on the performance of N2G.

We design two variants of N2G. One is called N2GD , which cancels detail destruction. The other is called N2GU , which cancels the update delay. These networks are trained and tested on the dataset of Gaussian noise, and the results are reported in Table 3. Without detail destruction, the noisy image xs synthesized by N2G is unrealistic, which leads to poor performance of the noise removal module. On the other hand, due to the cancellation of update delay, the gradient similarity loss Lgrad has a negative effect on the noise removal module, thus damaging the denoising results. The combination of these two techniques can achieve excellent results of noise removal and noise extraction.

3.3 Medical Image Denoising

Computed tomography (CT) provides critical clinical information, but there are potential risks induced by X-ray radiation, such as cancer diseases and genetic damages. Given these risks, reducing the radiation dose as much as possible has become a trend in CT-related research. However, the reduction of radiation dose is associated with the increase of noise and artifacts in the reconstructed image, which may adversely affect subsequent diagnosis. Here, we show the application of Noise2Grad on low-dose CT denoising.

The training dataset is an authorized clinical low-dose CT dataset, which was used for the 2016 NIH-AAPM-Mayo Clinic LDCT Grand Challenge. 2 This dataset contains 5946 pairs of images with a slice thickness of 1mm. We divide them into 3 parts, 500 pairs as the test set, 2718 pairs as Dnoise , and the remaining 2718 pairs as Dclean . N2G is compared with the fully-supervised U-Net. Results are shown in Figure 7 and Table 4. For low-dose CT denoising, the denoising performance of our N2G is close to the fully supervised U-Net. Our denoising network cleanly removes noise and restores high-quality images.

2 https://www.aapm.org/GrandChallenge/LowDoseCT/

4 Conclusion

We proposed Noise2Grad, a novel algorithm that uses unpaired noisy and clean images to train denoising CNNs. The core idea of N2G is to extract noise from noisy images, and then superimpose it on other clean images to synthesize paired training data. To facilitate noise extraction, we use a noise removal module to eliminate the interference of the image background. Constrained by the image gradient, the noise removal module and the noise approximation module cooperate to extract noise finely. We demonstrate the effectiveness and wide applicability of Noise2Grad over multiple denoising tasks. Since Noise2Grad does not require precollected paired data and assumptions about noise statistics, it is a promising solution for many practical applications.

Acknowledgments

This work is supported in part by the National Key Research and Development Program of China (No.2020YFC2003900), in part by the National Natural Science Foundation of China under Grants U19B2031, 61971369, in part by Hong Kong Research Grants Council through Research Impact Fund under Grant R-5001-18.