{"type":"ScholarlyArticle","title":"Hierarchical Bilevel Learning with Architecture and Loss Search for Hadamard-based Image Restoration","authors":[{"name":"Guijing Zhu","slug":"guijing-zhu","given":"Guijing","family":"Zhu","affiliation":"School of Software Technology , Dalian University of Technology"},{"name":"Long Ma","slug":"long-ma","given":"Long","family":"Ma","affiliation":"School of Software Technology , Dalian University of Technology; Peng Cheng Laboratory"},{"name":"Xin Fan","slug":"xin-fan","given":"Xin","family":"Fan","affiliation":"International School of Information Science & Engineering , Dalian University of Technology"},{"name":"Risheng Liu","slug":"risheng-liu","given":"Risheng","family":"Liu","affiliation":"International School of Information Science & Engineering , Dalian University of Technology"}],"abstract":"In the past few decades, Hadamard-based image restoration problems (e.g., low-light image enhancement) attract wide concerns in multiple areas related to artificial intelligence. However, existing works mostly focus on heuristically defining architecture and loss by the engineering experiences that came from extensive practices. This way brings about expensive verification costs for seeking out the optimal solution. To this end, we develop a novel hierarchical bilevel learning scheme to discover the architecture and loss simultaneously for different Hadamard-based image restoration tasks. \r\n  More concretely, we first establish a new Hadamard-inspired neural unit to aggregate domain knowledge into the network design. Then we model a triple-level optimization that consists of the architecture, loss and parameters optimizations to deliver a macro perspective for network learning. Then we introduce a new hierarchical bilevel learning scheme for solving the built triple-level model to progressively generate the desired architecture and loss. We also define an architecture search space consisting of a series of simple operations and an image quality-oriented loss search space. \r\n  Extensive experiments on three Hadamard-based image restoration tasks (including low-light image enhancement, single image haze removal and underwater image enhancement) fully verify our superiority against state-of-the-art methods.","doi":"10.24963/ijcai.2022/245","url":"https://ijcai.org/proceedings/2022/245","pdf":"https://ijcai.org/proceedings/2022/0245.pdf","markdown":"https://ijcai.org/proceedings/2022/245.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/245","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":245,"pages":{"first":1757,"last":1764},"keywords":["Computer Vision: Computational photography","Computer Vision: Other"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986620","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984763","length_sec":277,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ancuti et al., 2012] Cosmin Ancuti, Codruta Orniana An- cuti, Tom Haber, and Philippe Bekaert. Enhancing un- derwater images and videos by fusion. In CVPR, pages 81-88, 2012.","title":"Enhancing underwater images and videos by fusion","authors":"Cosmin Ancuti, ;, Tom Haber, Philippe Bekaert","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bychkovsky et al., 2011] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. Learning photo- graphic global tonal adjustment with a database of in- put/output image pairs. In CVPR, pages 97-104, 2011.","title":"Learning photographic global tonal adjustment with a database of input/output image pairs","authors":"Vladimir Bychkovsky, Sylvain Paris, Eric Chan, Frédo Durand","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Zeyuan Chen, Yangchao Wang, Yang Yang, and Dong Liu. Psd: Principled synthetic-to-real de- hazing guided by physical priors. In CVPR, pages 7180- 7189, 2021.","title":"Psd: Principled synthetic-to-real dehazing guided by physical priors","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2020] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In CVPR, pages 2157-2167, 2020.","title":"Multi-scale boosted dehazing network with dense feature fusion","authors":"Dong","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Drews et al., 2016] Paulo LJ Drews, Erickson R Nasci- mento, Silvia SC Botelho, and Mario Fernando Montene- gro Campos. Underwater depth estimation and image restoration based on single images. IEEE CGA, 36(2):24- 35, 2016.","title":"Underwater depth estimation and image restoration based on single images","authors":"Drews","year":"2016","venue":"IEEE CGA","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu and Cao, 2020] Xueyang Fu and Xiangyong Cao. Un- derwater image enhancement with global-local net- works and compressed-histogram equalization. SPIC, 86:115892, 2020.","title":"Xueyang Fu and Xiangyong Cao. Underwater image enhancement with global-local networks and compressed-histogram equalization","authors":"Cao Fu","year":"2020","venue":"SPIC","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2017] Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, and Xinghao Ding. Two-step approach for single underwater image enhancement. In ISPACS, pages 789- 794, 2017.","title":"Two-step approach for single underwater image enhancement","authors":"Fu","year":"2017","venue":"ISPACS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Islam et al., 2020] Md Jahidul Islam, Youya Xia, and Ju- naed Sattar. Fast underwater image enhancement for im- proved visual perception. IEEE RAL, 5(2):3227-3234, 2020.","title":"Fast underwater image enhancement for improved visual perception","authors":"Md Jahidul Islam, Youya Xia, Junaed Sattar","year":"2020","venue":"IEEE RAL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2021] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE TIP, 30:2340-2349, 2021.","title":"Enlightengan: Deep light enhancement without paired supervision","authors":"Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, Zhangyang Wang","year":"2021","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] ChongYi Li, JiChang Guo, RunMin Cong, YanWei Pang, and Bo Wang. Underwater image enhance- ment by dehazing with minimum information loss and his- togram distribution prior. IEEE TIP, 25(12):5664-5677, 2016.","title":"Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior","authors":"Li","year":"2016","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019a] Chongyi Li, Chunle Guo, Wenqi Ren, Run- min Cong, Junhui Hou, Sam Kwong, and Dacheng Tao. An underwater image enhancement benchmark dataset and beyond. IEEE TIP, 29:4376-4389, 2019.","title":"An underwater image enhancement benchmark dataset and beyond","authors":"Li","year":"2019","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019b] Chuming Li, Xin Yuan, Chen Lin, Ming- hao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs: Automl for loss function search. In ICCV, pages 8410- 8419, 2019.","title":"Am-lfs: Automl for loss function search","authors":"Li","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Chongyi Li, Saeed Anwar, and Fatih Porikli. Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recognition, 98:107038, 2020.","title":"Underwater scene prior inspired deep underwater image and video enhancement","authors":"Li","year":"2020","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2022] Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhou. Autoloss-zero: Search- ing loss functions from scratch for generic tasks. In CVPR, 2022.","title":"Autoloss-zero: Searching loss functions from scratch for generic tasks","authors":"Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, Xizhou Zhou","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. In ICLR, 2018.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-based multi-scale network for image dehazing. In ICCV, pages 7314-7323, 2019.","title":"Griddehazenet: Attention-based multi-scale network for image dehazing","authors":"Xiaohong Liu, Yongrui Ma, Zhihao Shi, Jun Chen","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021a] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimiza- tion for learning and vision from a unified perspective: A survey and beyond. IEEE TPAMI, 2021.","title":"Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond","authors":"Liu","year":"2021","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2021b] Risheng Liu, Zi Li, Xin Fan, Chenying Zhao, Hao Huang, and Zhongxuan Luo. Learning de- formable image registration from optimization: perspec- tive, modules, bilevel training and beyond. IEEE TPAMI, 2021.","title":"Learning deformable image registration from optimization: perspective, modules, bilevel training and beyond","authors":"Liu","year":"2021","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2021c] Risheng Liu, Xuan Liu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A value-function-based interior-point method for non-convex bi-level optimiza- tion. In Marina Meila and Tong Zhang, editors, ICML, volume 139, pages 6882-6892, 2021.","title":"A value-function-based interior-point method for non-convex bi-level optimization","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2021d] Risheng Liu, Yaohua Liu, Shangzhi Zeng, and Jin Zhang. Towards gradient-based bilevel optimiza- tion with non-convex followers and beyond. NeurIPS, 2021.","title":"Towards gradient-based bilevel optimization with non-convex followers and beyond","authors":"Liu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021e] Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In CVPR, pages 10561-10570, 2021.","title":"Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement","authors":"Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2022a] Risheng Liu, Long Ma, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. Task-oriented convex bilevel optimization with latent feasibility. IEEE TIP, 2022.","title":"Task-oriented convex bilevel optimization with latent feasibility","authors":"Liu","year":"2022","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2022b] Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A general descent aggre- gation framework for gradient-based bi-level optimization. IEEE TPAMI, 2022.","title":"A general descent aggregation framework for gradient-based bi-level optimization","authors":"Liu","year":"2022","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lv et al., 2018] Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim. Mbllen: Low-light image/video en- hancement using cnns. In BMVC, volume 220, page 4, 2018.","title":"Mbllen: Low-light image/video enhancement using cnns","authors":"Lv","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ma et al., 2021] Long Ma, Risheng Liu, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. Learning deep context-sensitive decomposition for low-light image enhancement. IEEE TNNLS, 2021.","title":"Learning deep context-sensitive decomposition for low-light image enhancement","authors":"Long Ma, Risheng Liu, Jiaao Zhang, Xin Fan, Zhongxuan Luo","year":"2021","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ma et al., 2022] Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In CVPR, 2022.","title":"Toward fast, flexible, and robust low-light image enhancement","authors":"Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, Zhongxuan Luo","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Osher et al., 2005] Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and Wotao Yin. An iterative regular- ization method for total variation-based image restoration. Multiscale Modeling & Simulation, 4(2):460-489, 2005.","title":"An iterative regularization method for total variation-based image restoration","authors":"Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, Wotao Yin","year":"2005","venue":"Multiscale Modeling & Simulation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pan et al., 2020] Jinshan Pan, Jiangxin Dong, Yang Liu, Ji- awei Zhang, Jimmy Ren, Jinhui Tang, Yu-Wing Tai, and Ming-Hsuan Yang. Physics-based generative adversarial models for image restoration and beyond. IEEE TPAMI, 43(7):2449-2462, 2020.","title":"Physics-based generative adversarial models for image restoration and beyond","authors":"Jinshan Pan, Jiangxin Dong, Yang Liu, Jiawei Zhang, Jimmy Ren, Jinhui Tang, Yu-Wing Tai, Ming-Hsuan Yang","year":"2020","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Qin et al., 2020] Xu Qin, Zhilin Wang, Yuanchao Bai, Xi- aodong Xie, and Huizhu Jia. Ffa-net: Feature fusion at- tention network for single image dehazing. In AAAI, vol- ume 34, pages 11908-11915, 2020.","title":"Ffa-net: Feature fusion attention network for single image dehazing","authors":"Zhilin Xu Qin, Yuanchao Wang, Xiaodong Bai, Huizhu Xie, Jia","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Qu et al., 2019] Yanyun Qu, Yizi Chen, Jingying Huang, and Yuan Xie. Enhanced pix2pix dehazing network. In CVPR, pages 8160-8168, 2019.","title":"Enhanced pix2pix dehazing network","authors":"Qu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Rosenfeld, 1976] Azriel Rosenfeld. Digital picture process- ing. 1976.","title":"Digital picture processing","authors":"Azriel Rosenfeld","year":"1976","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shao et al., 2020] Yuanjie Shao, Lerenhan Li, Wenqi Ren, Changxin Gao, and Nong Sang. Domain adaptation for image dehazing. In CVPR, pages 2808-2817, 2020.","title":"Domain adaptation for image dehazing","authors":"Shao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tan et al., 2020] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object de- tection. In CVPR, pages 10781-10790, 2020.","title":"Efficientdet: Scalable and efficient object detection","authors":"Mingxing Tan, Ruoming Pang, Quoc V Le","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Uplavikar et al., 2019] Pritish M Uplavikar, Zhenyu Wu, and Zhangyang Wang. All-in-one underwater image en- hancement using domain-adversarial learning. In CVPRW, pages 1-8, 2019.","title":"All-in-one underwater image enhancement using domain-adversarial learning","authors":"Zhenyu Pritish M Uplavikar, Zhangyang Wu, Wang","year":"2019","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2019] Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. Under- exposed photo enhancement using deep illumination esti- mation. In CVPR, pages 6849-6857, 2019.","title":"Underexposed photo enhancement using deep illumination estimation","authors":"Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, Jiaya Jia","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wei et al., 2018] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. In BMVC, 2018.","title":"Deep retinex decomposition for low-light enhancement","authors":"Chen Wei, Wenjing Wang, Wenhan Yang, Jiaying Liu","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xu et al., 2020] Ke Xu, Xin Yang, Baocai Yin, and Ryn- son WH Lau. Learning to restore low-light images via decomposition-and-enhancement. In CVPR, pages 2281- 2290, 2020.","title":"Learning to restore low-light images via decomposition-and-enhancement","authors":"Ke Xu, Xin Yang, Baocai Yin, Rynson Wh Lau","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yang et al., 2020] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to per- ceptual quality: A semi-supervised approach for low-light image enhancement. In CVPR, pages 3063-3072, 2020.","title":"From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement","authors":"Yang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhang et al., 2019] Yonghua Zhang, Jiawan Zhang, and Xi- aojie Guo. Kindling the darkness: A practical low-light image enhancer. In ACM MM, pages 1632-1640, 2019.","title":"Kindling the darkness: A practical low-light image enhancer","authors":"Yonghua Zhang, Jiawan Zhang, Xiaojie Guo","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zoph and Le, 2016] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.","title":"Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}