A Comprehensive Survey on Image Dehazing Based on Deep Learning

1 Introduction

Images with clear visibility are required for computer vision tasks [Li et al., 2017a]. However, due to the absorption by floating particles contained in the environment, images taken in hazy days often suffer from quality degradation. With the aim of restoring color and details from distorted images, it is necessary to design an effective dehazing algorithm.

Dehazing methods based on deep learning are mainly divided into supervised ID and unsupervised ID. There is not too much work about semi-supervised dehazing [Li et al., 2020b]. In general, supervised methods require pairs of hazy images and haze-free images/transmission maps. The unsupervised methods often take unpaired images as the training set. There are three important factors in the research of supervised ID and unsupervised ID:

• Atmospheric scattering model (ASM) [McCartney, 1976; Nayar and Narasimhan, 1999; Narasimhan and Nayar, 2003] provides a theoretical basis for the research of image dehazing.

∗Corresponding Author

• Transmission map (Trans): In ASM, Trans is the part of light that is not scattered and reaches the camera. After measuring the depth information of the scene, Trans can be obtained by the ASM.
• Extra data: With the help of pre-trained models, dehazing algorithms have achieved better performances by minimizing the perceptual loss [Justin et al., 2016].

Based on the above factors, researchers have designed many effective dehazing algorithms. For the convenience of discussion, we further divide the supervised dehazing algorithms into two types: ASM-based and non-ASM-based. Details can be found in Table 1.

Goals of our survey. We aim to (i) summarize commonly used datasets and loss functions in day-time dehazing task;

(ii) provide a taxonomy for state-of-the-art DL-based dehazing algorithms and introduce the core techniques among various methods; (iii) point out open issues that would inspire further researches in ID tasks.

The differences between our survey and others. [Singh and Kumar, 2018; Li et al., 2017b; Xu et al., 2016] gave a detailed summary of the classic dehazing methods, including depth estimation, wavelet, enhancement, filtering, but they lack the detailed discussion of recent neural network models. [ Parihar et al., 2020] provides some investigations of supervised dehazing models, but it does not pay attention to the latest applications of unsupervised methods. [Banerjee and Chaudhuri, 2020] systematically introduced and grouped the existing nighttime ID methods [Zhang et al., 2020], but seldom mentioned the dehazing methods for the day-time. Our survey gives a comprehensive survey on the supervised and unsupervised DL-based dehazing models for the day-time.

2 RelatedWork

In this section, we will introduce the atmospheric scattering model (ASM) [McCartney, 1976; Nayar and Narasimhan, 1999; Narasimhan and Nayar, 2003], widely used datasets, metrics for evaluation, and loss functions in the research of ID.

2.1 Atmosphere ScatteringModel

The formula of ASM is as following:

where x represents the position of pixels and A stands for the global atmospheric light. I(x) denotes the hazy scene and J(x) is the haze-free image that the dehazing method expects to obtain. The t(x) means the medium Trans:

where β and d(x) represent the atmosphere scattering parameter and the depth of the scene, respectively. Meanwhile, it means that the Trans t(x) can be obtained through the depth map d(x), which is very critical for the synthesis of the dataset. If we can estimate t(x) and A, the formula of the dehazing image J(x) is

This ASM is widely used in the research of image dehazing [Cai et al., 2016a] and video dehazing [Li et al., 2018a; Cai et al., 2016b].

2.2 Datasets for Image Dehazing Tasks

In order to ensure the reliability of the data, the researchers mainly use two strategies to obtain paired hazy and haze-free images. One is synthesized based on the dataset containing depth annotation, such as the D-Hazy [Ancuti et al., 2016] that synthesized on the basis of NYU-Depth [Silberman et al., 2012] and Middlebury [Scharstein et al., 2014]. Similar datasets include HazeRD [Zhang et al., 2017] and RESIDE [Li et al., 2019]. The other kind of dataset does not contain depth information, such as I-HAZE [Ancuti et al., 2018c], O-HAZE [Ancuti et al., 2018a], Dense-HAZE [Ancuti et al., 2019], NH-HAZE [Ancuti et al., 2020a], which are produced by a professional haze generator. Based on these four datasets, researchers have proposed a variety of effective dehazing algorithms in the New Trends in Image Restoration and Enhance (NTIRE 2018-2020) challenge [Ancuti et al., 2018b; C.Ancuti et al., 2019; Ancuti et al., 2020b].

However, there is an inherent difference between synthetic/generated data and real data, which is called domain shift. The recently proposed MRFID [Liu et al., 2020] and BeDDE [Zhao et al., 2020] contain hazy and haze-free images collected in the real world, but its insufficient data makes it unsuitable for large-capacity dehazing models. We list the attributes of various datasets in Table 2.

2.3 Image QualityMetrics

The two most common full-reference metrics in dehazing research are Peak Signal to Noise Ratio (PSNR) and structural similarity index (SSIM) [Wang et al., 2004]. Some dehazing researchers use CIEDE2000 [Sharma et al., 2005] to evaluate the color difference between the dehazed image and the ground-truth clean image.

2.4 Loss Function

For convenience, this section introduces the commonly used loss functions of supervised and unsupervised dehazing models. The widely used pixel-wise loss functions for dehazing research are L1 loss and L2 loss, which require fewer operations in the optimization process. As a metric of the dehazing model, SSIM is also used as a loss function in the optimization process [Dong et al., 2020b]:

However, neither L1/L2 loss nor SSIM loss considers the special attributes of the human visual system in the process of perceptual evaluation. Recent researches use pretrained classification neural networks to calculate the perceptual loss [Engin et al., 2018; Cong et al., 2020] in the feature space ψ(·):

Perceptual loss provides a way to use external data to assist the training of dehazing models as shown in Table 1, but we think it still needs further research.

3 Supervised Dehazing Algorithms

According to whether the ASM is used or not, the supervised dehazing methods can be divided into two groups as shown in Figure 1. The first group is to combine the ASM with the convolutional neural network to estimate the parameters for the dehazing process. This physical-model-based dehazing algorithm usually needs to utilize the depth information and the airlight value (black arrows in Figure 1). The second way is to directly learn the mapping from hazy to haze-free images (green arrow in Figure 1), which usually requires pairs of hazy and haze-free images.

3.1 Dehazing Based on the ASM

The dehazing algorithm based on deep learning and ASM needs to estimate two parameters: (1) Trans t(x), (2) atmospheric light value A. A variety of deep learning strategies have been successfully applied to the process of parameter estimation, such as multi-scale [Ren et al., 2016], pyramid network [Zhang and Patel, 2018], monocular estimation [Pang et al., 2020]. According to the different types of input data, we divide the supervised model based on ASM into monocular dehazing and binocular dehazing.

Monocular Dehazing

For simplicity, we further divide the monocular dehazing model into three types.

The first category uses a deep neural network to estimate the t(x) and uses prior assumptions to estimate A. Strictly, this kind of method is not an end-to-end dehazing model. Taking DehazeNet [Cai et al., 2016a] as an example, the L2 loss is used to estimate the Trans as following:

where m is the number of data in the training set. Another important parameter A in the ASM is estimated using 0.1% darkest pixels of t(x). Then, the final dehazed result J(x) is jointly determined by t(x) and A:

[Ren et al., 2020] also uses a trainable end-to-end approach to design a dehazing network via multi-scale convolutional neural networks with holistic edges. Compared with non-datadriven estimation methods [Zhu et al., 2015; He et al., 2010], these CNN based methods can obtain sharp-edged Trans.

The second category uses multiple neural networks to jointly estimate the t(x) and A. Densely connected pyramid dehazing network (DCPDN) [Zhang and Patel, 2018] was proposed to jointly learn the t(x) and A. In particular, DCPDN used a U-shape network to predict a 2D atmospheric light map with the same size as the image to be processed. The optimization process of joint discriminator is as following:

DCPDN is an end-to-end physical dehazing model by fully embedding the ASM into the overall optimization framework.

The third category is to re-formulate the ASM, so as to avoid the cumulative error caused by the estimation of t(x) and A. AOD-Net [Li et al., 2017a] integrates the process of parameter estimation into a simple framework through the reformulated ASM. According to the original ASM that shows in (1), the dehazing image J(x) is

In order to avoid the estimation of t(x) in the ASM, AODNet proposed the K(x), an intermediate parameter without physical meaning:

Next, the acquisition of the clear image J(x) only requires a single parameter K(x):

The experiment of AOD-Net has proved that reliable dehazing performance can still be obtained without separate estimation of t(x) and A. Similar ideas are adopted by FAMEDNet [Zhu et al., 2018; Zhang and Tao, 2020; Dong and Pan, 2020]. Since the collection of the t(x) of an image is a very time-consuming and expensive work, exploring new ways of using ASM is a hot topic that can effectively simplify the dehazing process.

Binocular Dehazing

The dehazing methods based on binocular images usually use left and right views as joint input, rather than a monocular image. [Song et al., 2019] proposed a multi-task network (SSMD-Net) that can join estimate clear images and disparity with the help of stereo hazy image pairs. SSMDNet has proved that stereo matching and ID tasks can be synergistically formulated by incorporating the depth from t(x) of the scene into the matching task. The limitation of SSID-Net is that it can only estimate the left haze-free images. Different from SSMD-Net, the recent proposed BidNet [Pang et al., 2020] can produce binocular dehazed image pairs. When estimating the stereo Trans, BidNet designed the Stereo Transformation Module (STM) to transform the depth information. For left input Fl ∈ RB×C×H×W and right input Fr ∈ RB×C×H×W, 1 × 1 convolutions (W l θ , W rψ and W r γ ) are used to get corresponding embeddings. The loss function of the Trans estimation process is composed of the left and right views:

where ˆ tl/ˆ tr denotes the estimated left/right Trans, ˆ trl/ˆ trr means the estimated refined left/right Trans, and tl/tr stands for the ground truth left/right Trans. [Song et al., 2020] combined the attention mechanism with binocular tasks and adapted the model to real-world images through selfsupervised distillation. Although the binocular dehazing method has reliable performances, it further enhances the dependence of the dehazing model on datasets.

3.2 Dehazing without the ASM

Some works focus on the design of dehazing models that do not rely on the estimation of t(x) and A in the ASM. For the direct learning of clear images, researchers have explored a variety of effective deep learning methods, such as attention mechanism [Qin et al., 2020], knowledge distillation [Hong et al., 2020], domain adaptation [Shao et al., 2020]. Here, we introduce their core ideas and contributions.

Multi-input Fusion

Without using the atmospheric scattering model, GFN [Ren et al., 2018] found two phenomena that affect the dehazing process: (i) the color of the image is sometimes distorted under the influence of atmospheric light, (ii) the visibility of the scene is reduced due to the scattering and attenuation. Based on these two observations, GFN proposed the idea of fusing three inputs: white balanced Iwb, contrast enhanced Ice, and gamma corrected Igc. The clear image J is determined by the combination of three inputs:

where Cwb, Cce, and Cgc denote the confidence maps for gating the three inputs, and ◦ means element-wise multiplication. The strategy of multi-input fusion provides a new idea of using hazy inputs, which is different from data enhancement (rotation, cropping, etc.).

Attention for Dehazing

Through theoretical analysis, the FFA-Net [Qin et al., 2020] proposed that channel-wise and pixel-wise features should not be treated equally when using CNN for feature extraction. Based on this observation, FFA-Net designed a feature attention (FA) module that includes the channel attention and pixel attention. First, global average pooling Hp is used to obtain global spatial information:

where Xc(i, j) stands for the c-th channel at position (i, j). Then, the channel attention output CAc is obtained as following:

where δ and ρ represent sigmoid and ReLU, respectively. The purpose of pixel attention PA is to pay more attention to areas such as thick-hazed pixels and high-frequency:

The Grid-DehazeNet [Liu et al., 2019] contains three subprocesses: pre-processing, backbone, and post-processing. The backbone module uses attention-based multi-scale estimation to learn features of the output of pre-processing. FFANet and Grid-DehazeNet prove that the attention mechanism has a reliable ability in the process of feature extraction and image restoration, which brings an effective strategy for dehazing research that does not use the ASM.

Generative Adversarial Networks (GAN) for Dehazing

The recent proposed enhanced pix2pix dehazing network (EPDN) [Qu et al., 2019] is based on the visual perception global-first theory. It contains three parts: the multiresolution generator, the multi-scale discriminator, and the enhancer. Different from the basic GAN [Goodfellow et al., 2014], the adversarial loss of EPDN includes the optimization process of two discriminators.

[Li et al., 2018b] also utilized the high-quality image generation capabilities of GAN. However, the training process of GAN may be unstable, and the adversarial loss is not easy to converge. Regarding the direct application of GAN in the field of image dehazing, more research is needed.

Knowledge Distillation for Dehazing

Knowledge distillation technology has been widely used in the research of classification and segmentation and it has been proven that knowledge transfer can be achieved. However, the teacher and student network of classification and segmentation often deal with the same task, which is not suitable for dehazing. The KDDN [Hong et al., 2020] set the teacher network (T) as the image reconstruction task, and made the student network (S) imitate this process. Combined with other solutions, KDDN bridges the gap between image dehazing and knowledge distillation. The unsupervised loss is used to constrain the features of each layer in the teacher network for the haze-free image:LT = ||J - T (J)||1. (18)

With the aim of domain transfer, KDDN combines three loss functions to train the S-network:

where Lr and Lp denote the reconstruction loss and the perceptual loss, respectively. Lrm is the representation mimicking loss which is designed for mimicking the representation of the haze-free domain. λp and λrm are the weight factors. KDDN explored the process of migrating knowledge from T - network to S-network for dehazing.

Domain Adaptation for Dehazing

Due to the existence of domain shift, the generalization ability of the dehazing model trained on synthetic data cannot be guaranteed when it is applied to real world hazy images. The DAID [Shao et al., 2020] proposed a domain adaptation method, which was realized by the joint training of an image translation module and two image dehazing modules. The image translation modules GS→R and GR→S are responsible for the translation between the synthetic domain and the real domain. With the help of an image-level discriminator Dimg R and a feature-level discriminator Dfeat R , the adversarial loss of the conversion process is calculated as following:

The domain adaptation method designed by DAID makes full use of the synthetic data for the dehazing task. However, the training of the dehazing network is a combination of unsupervised and supervised processes. The domain adaptation dehazing with the completely unsupervised training process is a topic worthy of further research.

4 Unsupervised Dehazing Algorithms

Although supervised dehazing algorithms have achieved excellent performances, they are highly dependent on paired data. In outdoor scenes that include grass, sky or water, it is difficult to ensure the correspondence between the content of the scene with and without haze. Therefore, the unsupervised dehazing methods are more suitable for real-world dehazing task, which can effectively reduce the cost of data acquisition. In this section, we introduce four types of representative unsupervised dehazing models as shown in Figure 2.

4.1 Two Domains

In order to avoid the usage of paired data, the Cycle-Dehaze network [Engin et al., 2018] was proposed to learn the bidirectional mapping functions between the hazy domain and haze-free domain. The generator G is responsible for learning the mapping from hazy images to haze-free images, and the training purpose of generator F is to obtain the mapping in the opposite direction. Randomly sampling x and y from the hazy image domain X and the haze-free image domain Y , the Cycle-Dehaze uses the (denoted as ψ(·)) to calculate cyclic perceptual consistency loss:

The overall loss function of Cycle-Dehaze is composed of cyclic perceptual consistency loss Lcyc-p and CycleGAN's [Zhu et al., 2017] loss function Lcyc-GAN. CycleDehaze provides a bridge between ID task and image-toimage translation within two domains as shown in Figure 2 (a). Similar methods [Huang et al., 2019; Liu et al., 2020] based on CycleGAN have been proposed, which adopted a CNN-based atmospheric degradation model to synthesize hazy images. In addition, DDN [Yang et al., 2018] used physical-model-based disentanglement and reconstruction mechanism for unsupervised training. First, the distorted image is disentangled into three factors: the clear image, t(x), and A, by three generators. Then, the three factors are combined to reconstruct the hazy input by the ASM.

4.2 Multiple Domains

Most deep learning dehazing methods generally treat hazy images of different levels as the same image domain, without considering the domain difference between multiple haze densities. The discrete haze level dehazing network (DHLDehaze) [Cong et al., 2020] processed multiple haze level domains from the perspective of multi-domain image-toimage translation as shown in Figure 2 (b). First, sample Xori = {x(1) ori , x(2) ori , . . . , x(m) ori } with corresponding labels Lori = {l(1) ori , l(2) ori , . . . , l(m) ori }. Through (22) and (23), classification loss and adversarial loss in the source domain can be obtained

The generation of the target domain images require Ltar = {l(1) tar , l(2) tar , . . . , l(m) tar } and the images of the source domain. DHL-Dehaze designed a multi-scale generator (MST) to produce high-quality images:

Then, the classification and adversarial loss of the target domain image can be obtained by using the similar process as (22) and (23). Through the adversarial training process between the source domain and the target domain, the hazy image can achieve the translation of haze level without changing the content of the scene. Furthermore, DHL-Dehaze studied the dehazing result of the same model under different haze densities.

4.3 Unsupervised Loss

The cyclic consistency loss used in the unsupervised training of cyc-Dehaze [Engin et al., 2018] and DHL-Dehaze [Cong et al., 2020] allows the model to learn the mapping for ID task without using paired data. Further, Deep-DCP [Golts et al., 2020] proposed to use only hazy images as input to update the parameters of the model, whose training process is to minimize the dark channel prior (DCP) [He et al., 2010] energy function. The dark channel is defined by image patch Ω(x):

Then, the following formula can be used to obtain the predicted Trans:

The energy function can be expressed as following:

The DCP loss designed by Deep-DCP can easily cooperate with different CNN architectures. The exploration of unsupervised loss is a valuable research direction.

4.4 Zero-shot Dehazing

Since unsupervised models always require sufficient data for training, zero-shot image dehazing (ZID) [Li et al., 2020a] tries to minimize the dependence on the number of images. ZID has the characteristics of zero-shot while being unsupervised, that is, it only needs to learn and infer about a single hazy image. ZID contains three sub-networks fJ(·) (JNet), fT (·) (T-Net), and fA(·) (A-Net), which are used to estimate the dehazed image, Trans, and atmospheric light, respectively. The overall loss function is as following:

The purpose of LRec loss is to disentangle the hazy image I(x) through the reconstruction process. The acquisition of atmospheric light A requires disentangled atmospheric light fA(x) and Kullback-Leibler (KL) divergence. For J-Net, the unsupervised channel loss LJ is used to decompose the clear image J(x). The limitation of ZID is that the inference speed for a single hazy image is relatively slow, because it requires separate training for each input.

5 Challenges and Opportunities

The current dehazing method has achieved good performances. Here, we attempt to highlight the problems that still exist and the open issues that would inspire the future research.

• Problems with current ASM: Although the current ASM has proven to be effective, it is only an approximation of the process of haze formation. In addition, [Ju et al., 2021] discovered that ASM-based dehazing methods often lead to very dim results due to intrinsic limitation, and proposed enhanced ASM (EASM). In order to design a better dehazing method, a more complete study of the ASM is worthwhile.
• Shift between the real domain and synthetic domain: (i) Due to the difficulty of collecting real-world data, training models on synthetic data and effectively migrating to the real world is an important topic. (ii) It is a challenging research to explore more realistic haze synthesis algorithms. (iii) Some works (MRFID/BeDDE) have tried to construct real-world datasets containing pairs of hazy and haze-free images, but their data volume is only 200+. Large and reliable datasets with paired real-world data are in urgent need.
• Computational efficiency and newmetrics: Since dehazing models are often used as pre-processing modules for high-level computer vision tasks, we must find a balance between the number of parameters, inference time, and quantitative performance. [Zhang and Tao, 2020] briefly discussed the computational efficiency of 14 dehazing methods. Future research can try to design a new evaluation metric that can comprehensively consider the running speed, model size, and dehazing quality.
• External data and pre-trained models: Many studies have shown that both supervised and unsupervised dehazing methods can improve the quality of dehazed images with the help of perceptual loss. However, there is no detailed discussion and proof of how external data and pre-trained models improve the performance of models. For example, do different types of pre-training models have different effects on performances?
• The haze density that the dehazing algorithm can handle: Most papers have discussed the model's effect on light and medium haze. Intuitively, the higher the density, the lower the quality of the dehazed image obtained by the dehazing model.

6 Conclusion

This paper provides a comprehensive survey of supervised and unsupervised image dehazing algorithms. We introduced the dependency relationship between the design of the dehazing model and specific data types. Finally, we pointed out the issues worthy of attention in the future of dehazing research.

Acknowledgments

This work was supported in part by the Fundamental Research Funds for the Central Universities under Grant 2242021R10097 and 202113037.