--- title: "Hierarchical Bilevel Learning with Architecture and Loss Search for Hadamard-based Image Restoration" authors: - "Guijing Zhu" - "Long Ma" - "Xin Fan" - "Risheng Liu" doi: "10.24963/ijcai.2022/245" url: "https://ijcai.org/proceedings/2022/245" pdf: "https://ijcai.org/proceedings/2022/0245.pdf" venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence" year: 2022 pages: "1757-1764" track: "Main Track" keywords: - "Computer Vision: Computational photography" - "Computer Vision: Other" --- # Hierarchical Bilevel Learning with Architecture and Loss Search for Hadamard-based Image Restoration Guijing Zhu, Long Ma, Xin Fan, Risheng Liu *Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 1757-1764. DOI: [10.24963/ijcai.2022/245](https://doi.org/10.24963/ijcai.2022/245) ## Abstract In the past few decades, Hadamard-based image restoration problems (e.g., low-light image enhancement) attract wide concerns in multiple areas related to artificial intelligence. However, existing works mostly focus on heuristically defining architecture and loss by the engineering experiences that came from extensive practices. This way brings about expensive verification costs for seeking out the optimal solution. To this end, we develop a novel hierarchical bilevel learning scheme to discover the architecture and loss simultaneously for different Hadamard-based image restoration tasks. More concretely, we first establish a new Hadamard-inspired neural unit to aggregate domain knowledge into the network design. Then we model a triple-level optimization that consists of the architecture, loss and parameters optimizations to deliver a macro perspective for network learning. Then we introduce a new hierarchical bilevel learning scheme for solving the built triple-level model to progressively generate the desired architecture and loss. We also define an architecture search space consisting of a series of simple operations and an image quality-oriented loss search space. Extensive experiments on three Hadamard-based image restoration tasks (including low-light image enhancement, single image haze removal and underwater image enhancement) fully verify our superiority against state-of-the-art methods. ## Full text ## 1 Introduction Image restoration with deep learning methods has developed rapidly in recent years, and Convolutional Neural Networks (CNNs) have been used widely in many image restoration tasks such as low-light image enhancement (LLIE), single image haze removal (SIHR) and underwater image enhancement (UIE) [Ma et al., 2022; Ma et al., 2021; Qin et al., 2020; Fu and Cao, 2020]. The performance of CNNs largely depends on the network architecture and training loss. In order to obtain networks with high performance, the automatic search of these two components has been explored recently. ∗Corresponding author. ## 1.1 RelatedWork Network Architecture Search. Great progress has been made in designing Neural Architecture Search (NAS) [Liu et al., 2021a; Liu et al., 2021e]. Early attempts [Zoph and Le, 2016] used evolutionary algorithms to optimize the architectures and parameters of the networks. Another approach [Tan et al., 2020] used reinforcement learning to train a meta-controller to find architectures. However, using evolutionary algorithms and reinforcement learning required a large amount of computations and made the search process more inefficient. Recently, differentiable search methods [Liu et al., 2018] were proposed. They converted the super-network into a differentiable form by introducing architectural parameters which could be optimized by gradient descent and the search cost could be reduced largely. The above algorithms have achieved competitive performance in various tasks. However, these existing methods tend to obtain unsatisfactory visual results in complicated scenarios. The main reason is that although these methods make use of NAS technique, they lacked physical constraints in principle. As a result, the architectures searched by the aforementioned methods often failed when meeting with complex degraded images because they couldn't capture the inherent image structure without the help of principle prior knowledge. Training Loss Search. Training loss is a critical component affecting performance. And most existing methods employed hand-craft losses with human experience. ℓ2 loss is a popular training loss in image restoration problem. However, it often brings about blurry in the images. Perceptual loss [Johnson et al., 2016] calculates ℓ2 distance between the activations of the hidden layers of a pre-trained network. However, using this loss could result in a large misalignment between training loss and evaluation metrics. So, many works combine multiple losses with carefully designed trade-off weights for complementation [Lv et al., 2018]. To find a better loss combination or form, training loss search has also aroused great interest very recently [Liu et al., 2021b; Li et al., 2019b; Li et al., 2022]. A cross-entropy based loss space was proposed in face recognition [Li et al., 2019b]. It conducted the search process including a set of cross-entropy losses for face recognition. But the types of candidate losses were too simple and of the same type. And it brought about a large misalignment between training loss and target metrics. The work in [Li et al., 2022] designed an elementary search space which was composed of primitive mathematical operators to accommodate the heterogeneous tasks and evaluation metrics. However, it abandoned the current existing loss evaluation system completely. More importantly, the searched combination result of these operators changed with randomness and the search space was too sparse, which may lead to unpleasing results. By contrast, our loss space is image quality-oriented and more relevant to the evaluation metrics. As a result, it is suitable for a lot of image restoration tasks. Figure 1: Hadamard-based image restoration. We present a Hadamard-based model which searches network architecture and training loss for restoring a given image captured on different Hadamard-based tasks such as (a) Low-light Image Enhancement (LLIE), (b) Single Image Haze Removal (SIHR), and (c) Underwater Image Enhancement (UIE). ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000000_575a357537c631c833d3bc8911dbe61406f1ec7cbb766eec7d3bc2cf6cfa69a4.png) ## 1.2 Our Contributions To address the above issues, we propose a novel hierarchical bilevel learning scheme with architecture and loss search for Hadamard-based image restoration. A Hadamard-inspired neural unit is firstly built to convert the domain knowledge into the network layer. Then we construct a novel triple-level modeling for architecture and loss search by characterizing the latent connection among loss, architecture and parameters. Considering the difficulty of solving the triple-level model, we define a new hierarchical bilevel learning scheme with an approximate solving process. A series of experiments verify our effectiveness on different tasks and the result is shown in Figure 1. In summary, our contributions can be concluded as - We develop a new Hadamard-inspired neural unit to integrate task-related domain knowledge into the network design to provide performance support for different Hadamard-based image restoration tasks. - We supply a general triple-level modeling perspective for joint architecture and loss search. It can accurately depict the latent correspondence between loss, architecture, and parameters. - To solve the triple-level model, we propose a hierarchical bilevel learning scheme that can alternatively estimate the architecture and loss to progressively generate the desired solutions. ## 2 The ProposedMethod In this section, we first construct our Hadamard-inspired neural unit. And then, we establish a new triple-level modeling for learning architecture and training loss. After that, we propose a stage-wise learning strategy for solving the built triplelevel model. Finally, we introduce the architecture search space and image quality-oriented loss search space. ## 2.1 Hadamard-inspired Neural Unit A set of image degradations can be represented as clear images with a degradation factor in Hadamard product form. And it can be formulated as y = u ⊗ v, where y denotes the degraded image, u denotes the original image and v denotes the degradation factor. ⊗ represents the element-wise multiplication. And u = y ⊘ v can be derived by performing inverse process, where ⊘ represents the element-wise division. By applying this rule to feature layer, we establish a Hadamard-inspired Neural Unit (HNU) to perform taskrelated domain knowledge. The designed HNU is consists of two feature enhancement blocks and is used for estimating u and v to realize degradation removal progressively. We cascade multiple HNUs and embed them into feature convert modules to form the optimization process of the model. The l-th intermediate HNU can be formulated as for the l-th block HNUl, the feature F lu obtained from the previous block (the F 1u(l = 1) is the feature generated by FCM) is inputted into two feature enhancement blocks Ru and Rv which will be searched to get F l+1 ˆu and F l+1 ˆv . Then the intermediate enhanced feature F l+1 u is generated by executing the element-wise division ⊘ on F l+1 ˆu andF l+1 ˆv . Themultiple cascaded HNUs framework is built by applying this equation. We design a Feature Convert Module (FCM) for conversion between image and feature domain. The whole framework is generated by embedding them into FCMs. The component of feature convert module also needs to be searched. The search space of the above modules can be found in Section 2.4. The overall framework is shown in Figure 2. ## 2.2 Triple-Level Modeling The crucial factors of performance for learning-based methods lie in how to define the architecture and training loss. As described above, we have clearly introduced a new block for Hadamard-based image restoration. Here we provide a general perspective for depicting the overall learning process. The formulation can be written as Figure 2: The overall network architecture. ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000001_562fbeb41978493cb137c2d371a6b679ab1f8949cb639df4fc5503c3b198c96f.png) where where β represents the hyper-parameters (including trade-off weights of the losses and inner parameters when calculating a specific loss) that needs to be defined in the loss space B. The variable α represents the desired architecture determined in the defined search space A. The variable ω denotes the network parameters related to α. The loss function LC val represents the classification-driven loss used on validation dataset. The loss LQ val and LQ tr denote the image quality-oriented loss 1 used on validation and training dataset, respectively. The above triple-level modeling actually builds an explicit relationship among training loss, network architecture, and network parameters. To be concrete, both of the loss and the architecture play a decisive role in the network parameters, i.e., the network parameters ω is the variable for the loss and architecture search. When searching the architecture, the adopted loss function directly dictates the form of the searched architecture, that is, the architecture α becomes the variable for the optimization of training loss. From above, the selection of the validation loss is the key to the success of searching training loss and should be designed carefully. However, the image quality-oriented losses (such as VGG and LPIPS loss) are employed in our loss space and can't be used as validation loss for search. So which loss to use as the validation loss is the key point. Considering how to evaluate whether an image is clear or not, we find that classification network is able to distinguish the image from its degraded form, which is related to the image restoration tasks. As a result, we employ a classification loss as our validation loss to estimate the performance of the searched training loss more accurately. What's more, by setting the degraded image as the negative sample and setting the clear image as the positive sample, the network can extract abstract representation and high level information of natural images. This information is helpful for generating visually pleasing results. More specifically, for each Hadamard-based image restoration task, we train an image classifier to distinguish whether the image is the clear one or not. And then we use the classification loss as the validation loss to search training loss. The specially trained classifier can capture more task-related information, which could evaluate the searched losses more accurately. 1 The specific forms about it can be found in Section 2.4. Table 1: The overall search space in the search phase for training loss. u is the restored image and u⋆ is the ground truth. | Name | Expression | Variable | |---------------|---------------------------------------------------|------------| | La: ℓ1 | λa ∗ ‖u⋆ - u‖1 | λa | | Lb: ℓ2 | λb ∗ ‖u⋆ - u‖2 | λb | | Lc: Color | λc ∗ ∑i ∠((u⋆ )i , (u)i ) | λc | | Ld: SSIM | λd ∗ l(u⋆ , u)α · c(u⋆ , u)β · s(u⋆ , u)γ | λd | | Le: LPIPS | λe ∗ ∑l ∑ h,w ∥∥∥wl⊙(F l (u⋆ )-F l (u))∥∥∥22 HlWl | λe | | Lf : VGG | λf ∗ ∑ n=1 ∥∥∥Vl (u⋆ )-Vl (u)∥∥∥1 Nl | λf , l | | Lg: TV | λg ∗ ((▽xu)2+(▽yu)2 ) H∗W | λg | | Lh: Laplacian | λh ∗ (▽2x u+▽2y u) H∗W | λh | ## 2.3 Hierarchical Bilevel Learning Scheme Actually, solving the above built triple-level model is a challenging issue because of the complex functional relationship. Here we propose a hierarchical bilevel solution scheme to convert the triple-level model into two bilevel models. Because bilevel optimization is more simple and has been researched many years compared to the triple-level problem [Liu et al., 2022b; Liu et al., 2022a; Liu et al., 2021d; Liu et al., 2021c]. By fixing the variable α when optimizing the β, we can obtain the following solution scheme where t represents the alternative number. The first equation indicates the optimization process of β. The classification loss LC val is used for validation. α is fixed as its initial value, and the solving process of β is converted into a bilevel problem involving the trainable variable β, ω and the fixed variable α. The second equation indicates the optimization process of α. The LQ val is obtained via the solution procedure of β. So the solving process of α is converted into another bilevel problem involving the trainable variable α and ω. What's more, in the solving process of the two bilevel problems, the gradients of weight parameters and architecture are computed using first-order approximation following standard differential NAS techniques [Liu et al., 2018]. ## 2.4 Search Space of Architecture and Loss Training Loss Search Space. We design an image qualityoriented losses set as the training loss search space. The candidate losses include ℓ1, ℓ2, color [Wang et al., 2019], SSIM, VGG [Johnson et al., 2016], LPIPS [Zhang et al., 2018], TV [Osher et al., 2005] and Laplacian [Rosenfeld, 1976] loss. For each loss, we define the weighting coefficient as the parameter to be optimized. What's more, the VGG loss has another kind of variable which is the selection of the hidden layers of the VGG network. We divide the layers of the VGG network into four groups in terms of feature scale, and the goal is to select one layer from each group. By employing a new variable for each group, we added the selection of VGG layers into loss search space. The detailed definition of each loss is shown in Table 1. Figure 3: Visual comparison among state-of-the-art LLIE methods on LOL dataset. ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000002_a7119eb6d31b9bd12774f579f3d1123276658b5f3ea91a48c216bb726133cb49.png) Table 2: Qualitative results of state-of-the-art methods and ours on the MIT-Adobe 5K and LOL datasets. | | Methods | MBLLEN | RetinexNet | DeepUPE | KinD | EnGAN | DRBN | FIDE | RUAS | Ours | |-----|-----------|----------|--------------|-----------|---------|---------|---------|---------|---------|---------| | MIT | PSNR | 17.4947 | 12.5591 | 20.8741 | 15.7552 | 15.1014 | 16.3702 | 16.2580 | 17.0056 | 23.2378 | | | SSIM | 0.6218 | 0.5673 | 0.6958 | 0.6318 | 0.6001 | 0.5967 | 0.6095 | 0.6182 | 0.8184 | | | LPIPS | 0.1787 | 0.2234 | 0.0937 | 0.1387 | 0.1218 | 0.1734 | 0.1769 | 0.1164 | 0.0519 | | LOL | PSNR | 18.6313 | 16.3076 | 11.8956 | 20.2552 | 17.6844 | 18.2622 | 21.5797 | 14.9663 | 24.1640 | | | SSIM | 0.7705 | 0.4362 | 0.4418 | 0.8382 | 0.6081 | 0.6688 | 0.7935 | 0.4986 | 0.8663 | | | LPIPS | 0.2161 | 0.4997 | 0.2793 | 0.1079 | 0.2975 | 0.1508 | 0.1307 | 0.2432 | 0.0751 | Architecture Search Space. According to different characteristics, we define search spaces for HNU and FCM respectively. The candidate operations include 1×1, 3×3 and 5×5 Convolution (1-C, 3-C and 5-C), 1×1 and 3×3 Residual Convolution (1-RC and 3-RC),3×3 Dilation Convolution with dilation rate of 2 (3-2-DC), 3×3 Residual Dilation Convolution with dilation rate of 2 (3-2-RDC), 1×1 and 3×3 Dense Convolution (1-DC and 3-DC), and Skip Connection (SC). The division info for HNU and FCM is shown in Figure 2. ## 3 Experimental Results ## 3.1 Implementation Details For LLIE task, we used MIT-Adobe 5K [Bychkovsky et al., 2011] and LOL Dataset [Wei et al., 2018]. We sampled 600 low-light images randomly from the MIT-Adobe 5K dataset for searching and training, and 100 images for testing. As for the LOL Dataset, we randomly sampled 690 image pairs for searching and training, and the remaining 99 pairs for testing. For SIHR task, we used the dataset proposed by [Pan et al., 2020]. And for UIE task, we sampled 712 image pairs from UIEB dataset [Li et al., 2019a] for searching and training and the remaining 178 image pairs for testing. All the experiments were performed on a PC with a single TITAN X GPU. We used ResNet-50 to train our classifier. In searching phase, the maximum epoch was 30, the batch size was 1 and the initial learning rate was 0.0005. The momentum was 0.9 and the weight decay was 0.0003. The initiate weight of each loss in loss search space was 1.0. As for the training phase (with the searched architecture and searched training loss), we set the learning rate as 0.0001 and the batch size was 1. Table 3: Quantitative SIHR result of state-of-the-art methods. | Methods | PSNR | SSIM | LPIPS | |---------------|---------|--------|---------| | DAD | 20.2702 | 0.8050 | 0.1438 | | EPDN | 20.6930 | 0.8232 | 0.0732 | | FFA | 22.6119 | 0.8830 | 0.0726 | | GridDehazeNet | 23.8632 | 0.8965 | 0.0677 | | MSBDN | 23.8819 | 0.8965 | 0.0677 | | PSD | 25.1914 | 0.9107 | 0.0675 | | PhysicsGAN | 24.4496 | 0.8806 | 0.0623 | | Ours | 27.0509 | 0.9370 | 0.0484 | Figure 4: Visual comparison among state-of-the-art SIHR methods on the real hazy image. ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000003_67e6c566abf6701171d4d04b900db75cf7ceaae8a9d492444ae9ac65cfe8573a.png) Table 4: Qualitative comparison of UIE task on UIEB dataset between our methods and state-of-the-art methods. | Methods | UDCP | OCM | TSA | Fusion | AIO | WaterNet | UWCNN | FGAN | GLN | Ours | |-----------|---------|---------|---------|----------|---------|------------|---------|---------|---------|---------| | PSNR | 11.6973 | 22.0781 | 14.2882 | 21.8875 | 13.3344 | 19.5888 | 17.5703 | 17.6067 | 21.0797 | 23.0009 | | SSIM | 0.5060 | 0.5838 | 0.4851 | 0.8200 | 0.4850 | 0.8238 | 0.6347 | 0.5880 | 0.7548 | 0.8630 | | LPIPS | 0.2635 | 0.2040 | 0.3187 | 0.1390 | 0.2851 | 0.1135 | 0.1911 | 0.2287 | 0.1179 | 0.0822 | Figure 5: Visual comparison among state-of-the-art UIE methods on UIEB dataset. ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000004_850283da6a769b9763bf53418f789b0465b215ef49ec52e6b66314ed3f025da0.png) ## 3.2 Comparison with State-of-the-arts To evaluate our method roundly, we adopted PSNR, SSIM and LPIPS as our evaluation metrics for three tasks. To fully demonstrate the superiority of our method, we compared our method with recently proposed state-of-the-art approaches on LLIE, SIHR and UIE tasks respectively. Evaluations on LLIE. The compared methods include MBLLEN [Lv et al., 2018], RetinexNet [Wei et al., 2018], DeepUPE [Wang et al., 2019], KinD [Zhang et al., 2019], EnGAN [Jiang et al., 2021], DRBN [Yang et al., 2020], FIDE [Xu et al., 2020] and RUAS [Liu et al., 2021e]. As shown in Figure 3, the compared methods such as DeepUPE and RUAS either couldn't remove noise or caused color shift. By contrast, our method obtained the highest performance in removing noise and enhancing the image and was superior to others in both the evaluation metrics and visual results. Evaluations on SIHR. The compared methods contain DAD [Shao et al., 2020], EPDN [Qu et al., 2019], FFA [Qin et al., 2020], GridDehazeNet [Liu et al., 2019], MSBDN [Dong et al., 2020], PSD [Chen et al., 2021] and PhysicsGAN [Pan et al., 2020]. The quantitative result was in Table 3 and the qualitative result was in Figure 4. As shown, most methods failed on the real hazy image. The reason is that these methods can't learn the inherent characteristics of the real hazy image. By comparison, our method was superior to others in both haze removal and detail recovery. Evaluations on UIE. The compared methods include UDCP [Drews et al., 2016], OCM [Li et al., 2016], TSA [Fu et al., 2017], Fusion [Ancuti et al., 2012], AIO [Uplavikar et al., 2019], WaterNet [Li et al., 2019a], UWCNN [Li et al., 2020], FGAN [Islam et al., 2020] and GLN [Fu and Cao, 2020]. As showed in Table 4 and Figure 5, our method gained the highest scores and could enhance the challenging underwater image well while others failed to recover clear details. ## 4 Algorithmic Analyses ## 4.1 The Effect of Training Loss Search To demonstrate the effect of training loss search. We conducted three groups of contrast experiments on LLIE task. (1): Employ MBLLEN as the basic model and its proposed loss as loss space. Use the experienced loss setting proposed by the author (MBLLEN) and compare it with the searched version (MBLLEN+). (2): Employ UNET as the basic model and our proposed loss space as loss space. Set the weights as 1.0 for each candidate (UNET) and compare it with the searched version (UNET+). (3): Using our searched architectures, set the weights as 1.0 for each candidate (Ours) and compare it with the searched version (Ours+). The result is shown in Figure 6. As showed, the searched version obtained a higher performance on both PSNR and SSIM than another. And our proposed method obtained the highest performance. ## 4.2 The Effect of Architecture Search We analyzed the performance among our model and different heuristically-designed architectures on LLIE task and they were trained using the searched loss. The result was shown in Table 5. As can be seen, the performance of the supernet and heuristically designed architectures was either unideal or couldn't reach the searched architectures. Briefly, these architectures may not effective enough. It is because these architectures did not integrate the task cues/principles. By comparison, our searched architecture realized the highest performance in all metrics. In a word, this experiment indicated the necessity of searching the architecture and the superiority of our searched architecture. Figure 6: Quantitative results between naively determined and searched training loss on LOL dataset. ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000005_ca32bd425ab7c9711ac22ceb6d0a6df69fd2d0038ee0313ad232a20226bebf7d.png) Table 5: Quantitative results between naively determined architectures and our searched architecture on LOL dataset. | Model | PSNR | SSIM | LPIPS | |----------|---------|--------|---------| | Supernet | 17.4412 | 0.7410 | 0.2447 | | 1-C | 21.3163 | 0.7708 | 0.1431 | | 3-C | 21.9472 | 0.8573 | 0.0947 | | 1-RC | 20.3591 | 0.6253 | 0.1938 | | 3-RC | 22.4196 | 0.8612 | 0.0816 | | Oursa | 22.9998 | 0.8186 | 0.0858 | | Ours | 24.1640 | 0.8663 | 0.0751 | ## 4.3 Analyzing the Search Strategy Actually, the search strategy is a decisive factor for the searched architecture. Here, we made an evaluation on the search strategy. Oursa: search training loss and architecture synchronously using the same classification loss LC val as the upper loss. In this way, architecture (α) and training loss (β) are searched by the same optimal goal at the same time. Ours: divide the triple-level problem into two bilevel problems. Specifically, firstly fix the architecture (α) as the supernet and use the classification loss (LC val ) to search training loss, then use this searched loss (LQ val ) to search the architecture. As shown in Table 5, the second strategy achieved higher performance and gained a large advantage. ## 4.4 The Searched Results of Different Tasks To better analyze our triple-level model for different tasks and explore the differences between tasks. We plotted the searched architectures and training loss of LLIE, SIHR and UIE tasks in Figure 7. For simplicity, we just selected one searched layer of the architectures from the HNU to show. It should be noted that the searched weights of TV and Laplacian loss are small to zero and don't show great differences among tasks. To focus on more useful information, we didn't draw them in the heatmap. And for easier observation, we softmax the searched loss weights of each task. As can be seen, the searched architectures were different in tasks. With the help of Hadamard principle, 1-C or 1-D was enough in this layer for LLIE and UIE task, while SC was searched in SIHR tasks. And the dominant loss also differed greatly in tasks, which were ℓ1, VGG and SSIM loss for LLIE, SIHR and UIE respectively. As shown in Figure 8, we showed the feature processed by the searched layer in second group of the VGG network. And present the 34-th, 42-th,51-th and 60-th channel feature to a colormap. As can be seen, the perceptual features required of SIHR and UIE were similar, while that of LLIE focused more on architectural characteristics. Figure 7: The searched losses and architectures of different tasks. ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000006_ec3f6968272f5b32a4413f6ecf967d66e02f1a55f84a1492b71b761877ff266c.png) Figure 8: Comparing features of different tasks processed by their searched VGG layers. The first to third rows are the inputs, the restored results of our method and the corresponding features. ![Image](https://ijcai.org/proceedings/2022/fulltext/245/image_000007_b8d51a849fe3c04e8e68247cf18618b37f728133b095885795c3677b7a137971.png) ## 5 Concluding Remarks We proposed a new framework to combine domain knowledge with a new search strategy to perform architecture and loss search for Hadamard-based image restoration. We first developed a Hadamard-inspired neural unit to explore taskrelated knowledge. And by cascading multiple units into feature convert modules, we can obtain the whole holistic structure of our enhancement network. And we designed an architecture search space and an image quality-oriented loss search space. Then we designed a hierarchical bilevel learning strategy to solve the triple-level problems including the architecture, loss and network parameters. Our experiments were performed on three image restoration tasks and we obtained new state-of-the-art results. In the future, we will apply our built triple-level model technique to different vision tasks. ## Acknowledgments This work is partially supported by the National Natural Science Foundation of China (Nos. 61922019, 61733002 and 62027826), the major key project of Peng Cheng Laboratory (No. PCL2021A12) and the Fundamental Research Funds for the Central Universities. ## References 1. [Ancuti et al., 2012] Cosmin Ancuti, Codruta Orniana An- cuti, Tom Haber, and Philippe Bekaert. Enhancing un- derwater images and videos by fusion. In CVPR, pages 81-88, 2012. 2. [Bychkovsky et al., 2011] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. Learning photo- graphic global tonal adjustment with a database of in- put/output image pairs. In CVPR, pages 97-104, 2011. 3. [Chen et al., 2021] Zeyuan Chen, Yangchao Wang, Yang Yang, and Dong Liu. Psd: Principled synthetic-to-real de- hazing guided by physical priors. In CVPR, pages 7180- 7189, 2021. 4. [Dong et al., 2020] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In CVPR, pages 2157-2167, 2020. 5. [Drews et al., 2016] Paulo LJ Drews, Erickson R Nasci- mento, Silvia SC Botelho, and Mario Fernando Montene- gro Campos. Underwater depth estimation and image restoration based on single images. IEEE CGA, 36(2):24- 35, 2016. 6. [Fu and Cao, 2020] Xueyang Fu and Xiangyong Cao. Un- derwater image enhancement with global-local net- works and compressed-histogram equalization. SPIC, 86:115892, 2020. 7. [Fu et al., 2017] Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, and Xinghao Ding. Two-step approach for single underwater image enhancement. In ISPACS, pages 789- 794, 2017. 8. [Islam et al., 2020] Md Jahidul Islam, Youya Xia, and Ju- naed Sattar. Fast underwater image enhancement for im- proved visual perception. IEEE RAL, 5(2):3227-3234, 2020. 9. [Jiang et al., 2021] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE TIP, 30:2340-2349, 2021. 10. [Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711, 2016. 11. [Li et al., 2016] ChongYi Li, JiChang Guo, RunMin Cong, YanWei Pang, and Bo Wang. Underwater image enhance- ment by dehazing with minimum information loss and his- togram distribution prior. IEEE TIP, 25(12):5664-5677, 2016. 12. [Li et al., 2019a] Chongyi Li, Chunle Guo, Wenqi Ren, Run- min Cong, Junhui Hou, Sam Kwong, and Dacheng Tao. An underwater image enhancement benchmark dataset and beyond. IEEE TIP, 29:4376-4389, 2019. 13. [Li et al., 2019b] Chuming Li, Xin Yuan, Chen Lin, Ming- hao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs: Automl for loss function search. In ICCV, pages 8410- 8419, 2019. 14. [Li et al., 2020] Chongyi Li, Saeed Anwar, and Fatih Porikli. Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recognition, 98:107038, 2020. 15. [Li et al., 2022] Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhou. Autoloss-zero: Search- ing loss functions from scratch for generic tasks. In CVPR, 2022. 16. [Liu et al., 2018] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. In ICLR, 2018. 17. [Liu et al., 2019] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-based multi-scale network for image dehazing. In ICCV, pages 7314-7323, 2019. 18. [Liu et al., 2021a] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimiza- tion for learning and vision from a unified perspective: A survey and beyond. IEEE TPAMI, 2021. 19. [Liu et al., 2021b] Risheng Liu, Zi Li, Xin Fan, Chenying Zhao, Hao Huang, and Zhongxuan Luo. Learning de- formable image registration from optimization: perspec- tive, modules, bilevel training and beyond. IEEE TPAMI, 2021. 20. [Liu et al., 2021c] Risheng Liu, Xuan Liu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A value-function-based interior-point method for non-convex bi-level optimiza- tion. In Marina Meila and Tong Zhang, editors, ICML, volume 139, pages 6882-6892, 2021. 21. [Liu et al., 2021d] Risheng Liu, Yaohua Liu, Shangzhi Zeng, and Jin Zhang. Towards gradient-based bilevel optimiza- tion with non-convex followers and beyond. NeurIPS, 2021. 22. [Liu et al., 2021e] Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In CVPR, pages 10561-10570, 2021. 23. [Liu et al., 2022a] Risheng Liu, Long Ma, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. Task-oriented convex bilevel optimization with latent feasibility. IEEE TIP, 2022. 24. [Liu et al., 2022b] Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A general descent aggre- gation framework for gradient-based bi-level optimization. IEEE TPAMI, 2022. 25. [Lv et al., 2018] Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim. Mbllen: Low-light image/video en- hancement using cnns. In BMVC, volume 220, page 4, 2018. 26. [Ma et al., 2021] Long Ma, Risheng Liu, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. Learning deep context-sensitive decomposition for low-light image enhancement. IEEE TNNLS, 2021. 27. [Ma et al., 2022] Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In CVPR, 2022. 28. [Osher et al., 2005] Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and Wotao Yin. An iterative regular- ization method for total variation-based image restoration. Multiscale Modeling & Simulation, 4(2):460-489, 2005. 29. [Pan et al., 2020] Jinshan Pan, Jiangxin Dong, Yang Liu, Ji- awei Zhang, Jimmy Ren, Jinhui Tang, Yu-Wing Tai, and Ming-Hsuan Yang. Physics-based generative adversarial models for image restoration and beyond. IEEE TPAMI, 43(7):2449-2462, 2020. 30. [Qin et al., 2020] Xu Qin, Zhilin Wang, Yuanchao Bai, Xi- aodong Xie, and Huizhu Jia. Ffa-net: Feature fusion at- tention network for single image dehazing. In AAAI, vol- ume 34, pages 11908-11915, 2020. 31. [Qu et al., 2019] Yanyun Qu, Yizi Chen, Jingying Huang, and Yuan Xie. Enhanced pix2pix dehazing network. In CVPR, pages 8160-8168, 2019. 32. [Rosenfeld, 1976] Azriel Rosenfeld. Digital picture process- ing. 1976. 33. [Shao et al., 2020] Yuanjie Shao, Lerenhan Li, Wenqi Ren, Changxin Gao, and Nong Sang. Domain adaptation for image dehazing. In CVPR, pages 2808-2817, 2020. 34. [Tan et al., 2020] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object de- tection. In CVPR, pages 10781-10790, 2020. 35. [Uplavikar et al., 2019] Pritish M Uplavikar, Zhenyu Wu, and Zhangyang Wang. All-in-one underwater image en- hancement using domain-adversarial learning. In CVPRW, pages 1-8, 2019. 36. [Wang et al., 2019] Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. Under- exposed photo enhancement using deep illumination esti- mation. In CVPR, pages 6849-6857, 2019. 37. [Wei et al., 2018] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. In BMVC, 2018. 38. [Xu et al., 2020] Ke Xu, Xin Yang, Baocai Yin, and Ryn- son WH Lau. Learning to restore low-light images via decomposition-and-enhancement. In CVPR, pages 2281- 2290, 2020. 39. [Yang et al., 2020] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to per- ceptual quality: A semi-supervised approach for low-light image enhancement. In CVPR, pages 3063-3072, 2020. 40. [Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018. 41. [Zhang et al., 2019] Yonghua Zhang, Jiawan Zhang, and Xi- aojie Guo. Kindling the darkness: A practical low-light image enhancer. In ACM MM, pages 1632-1640, 2019. 42. [Zoph and Le, 2016] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.