{"type":"ScholarlyArticle","title":"KUNet: Imaging Knowledge-Inspired Single HDR Image Reconstruction","authors":[{"name":"Hu Wang","slug":"hu-wang","given":"Hu","family":"Wang","affiliation":"School of CSE , University of Electronic Science and Technology of China , Chengdu , China"},{"name":"Mao Ye","slug":"mao-ye","given":"Mao","family":"Ye","affiliation":"School of CSE , University of Electronic Science and Technology of China , Chengdu , China"},{"name":"Xiatian Zhu","slug":"xiatian-zhu","given":"Xiatian","family":"Zhu","affiliation":"Surrey Institute for People-Centred Artificial Intelligence , CVSSP , University of Surrey , Guildford , UK"},{"name":"Shuai Li","slug":"shuai-li","given":"Shuai","family":"Li","affiliation":"School of Control Science and Engineering , Shandong University , Jinan , China"},{"name":"Ce Zhu","slug":"ce-zhu","given":"Ce","family":"Zhu","affiliation":"School of ICE , University of Electronic Science and Technology of China , Chengdu , China"},{"name":"Xue Li","slug":"xue-li","given":"Xue","family":"Li","affiliation":"School of ITEE , The University of Queensland , Brisbane , Australia"}],"abstract":"Recently, with the rise of high dynamic range (HDR) display devices, there is a great demand to transfer traditional low dynamic range (LDR) images into HDR versions. The key to success is how to solve the many-to-many mapping problem. However, the existing approaches either do not consider constraining solution space or just simply imitate the inverse camera imaging pipeline in stages, without directly formulating the HDR image generation process.  In this work, we address this problem by integrating LDR-to-HDR imaging knowledge into an UNet architecture, dubbed as Knowledge-inspired UNet (KUNet). The conversion from LDR-to-HDR image is mathematically formulated, and can be conceptually divided into recovering missing details, adjusting imaging parameters and reducing imaging noise. Accordingly, we develop a basic knowledge-inspired block (KIB) including three subnetworks corresponding to the three procedures in this HDR imaging process. The KIB blocks are cascaded in the similar way to the UNet to construct HDR image with rich global information. In addition, we also propose a knowledge inspired jump-connect structure to fit a dynamic range gap between HDR and LDR images. Experimental results demonstrate that the proposed KUNet achieves superior performance compared with the state-of-the-art methods. The code, dataset and appendix materials are available at https://github.com/wanghu178/KUNet.git.","doi":"10.24963/ijcai.2022/196","url":"https://ijcai.org/proceedings/2022/196","pdf":"https://ijcai.org/proceedings/2022/0196.pdf","markdown":"https://ijcai.org/proceedings/2022/196.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/196","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":196,"pages":{"first":1408,"last":1414},"keywords":["Computer Vision: Machine Learning for Vision","Computer Vision: Neural generative models, auto encoders, GANs","Computer Vision: Video analysis and understanding","Computer Vision: Applications","Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987007","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985150","length_sec":573,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2021a] Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, and Chao Dong. HDRUnet: Single image HDR re- construction with denoising and dequantization. In CVPR, pages 354-363, 2021.","title":"HDRUnet: Single image HDR reconstruction with denoising and dequantization","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021b] Xiangyu Chen, Zhengwen Zhang, Jimmy S Ren, Lynhoo Tian, Yu Qiao, and Chao Dong. A new journey from SDRTV to HDRTV. In ICCV, pages 4500-4509, 2021.","title":"A new journey from SDRTV to HDRTV","authors":"Chen","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2021] Shen Cheng, Yuzhi Wang, Haibin Huang, Donghao Liu, Haoqiang Fan, and Shuaicheng Liu. NBNet: Noise basis learning for image denoising with subspace projection. In CVPR, pages 4896-4906, 2021.","title":"NBNet: Noise basis learning for image denoising with subspace projection","authors":"Shen Cheng, Yuzhi Wang, Haibin Huang, Donghao Liu, Haoqiang Fan, Shuaicheng Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Demetris et al., 2018] Marnerides Demetris, Bashford-Rogers Thomas, Hatchett Jonathan, and Debattista Kurt. EXPNet:A deep convolutional neural network for high dynamic range expansion from low dynamic range content. In CGF, volume 37, pages 37-49. Wiley Online Library, 2018.","title":"EXPNet:A deep convolutional neural network for high dynamic range expansion from low dynamic range content","authors":"Demetris","year":"2018","venue":"CGF","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eilertsen et al., 2017a] Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K Mantiuk, and Jonas Unger. HDR image recon- struction from a single exposure using deep CNNs. ACM TOG, 36(6):1-15, 2017.","title":"HDR image reconstruction from a single exposure using deep CNNs","authors":"Eilertsen","year":"2017","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Eilertsen et al., 2017b] Gabriel Eilertsen, Rafal Konrad Mantiuk, and Jonas Unger. A comparative review of tone-mapping algo- rithms for high dynamic range video. In CGF, volume 36, pages 565-592. Wiley Online Library, 2017.","title":"A comparative review of tone-mapping algorithms for high dynamic range video","authors":"Eilertsen","year":"2017","venue":"CGF","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Froehlich et al., 2014] Jan Froehlich, Stefan Grandinetti, Bernd Eberhardt, Simon Walter, Andreas Schilling, and Harald Brendel. Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and hdr-displays. In Digital photography X, volume 9023, page 90230X. SPIE, 2014.","title":"Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and hdr-displays","authors":"Froehlich","year":"2014","venue":"Digital photography X","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gharbi et al., 2017] Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. Deep bilateral learning for real-time image enhancement. ACM TOG, 36(4):1- 12, 2017.","title":"Deep bilateral learning for real-time image enhancement","authors":"Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, Frédo Durand","year":"2017","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hasinoff et al., 2010] Samuel W Hasinoff, Frédo Durand, and William T Freeman. Noise-optimal capture for high dynamic range photography. In CVPR, pages 553-560. IEEE, 2010.","title":"Noise-optimal capture for high dynamic range photography","authors":"Frédo Samuel W Hasinoff, William T Durand, Freeman","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, pages 630-645. Springer, 2016.","title":"Identity mappings in deep residual networks","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2020] Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. Conditional sequential modulation for efficient global image re- touching. In ECCV, pages 679-695. Springer, 2020.","title":"Conditional sequential modulation for efficient global image retouching","authors":"Jingwen He, Yihao Liu, Yu Qiao, Chao Dong","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huo et al., 2014] Yongqing Huo, Fan Yang, Le Dong, and Vincent Brost. Physiological inverse tone mapping based on retina re- sponse. TVC, 30(5):507-517, 2014.","title":"Physiological inverse tone mapping based on retina response","authors":"Yongqing Huo, Fan Yang, Le Dong, Vincent Brost","year":"2014","venue":"TVC","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ibtehaz and Rahman, 2020] Nabil Ibtehaz and M Sohel Rahman. MultiResUNet: Rethinking the U-Net architecture for mul- timodal biomedical image segmentation. Neural Networks, 121:74-87, 2020.","title":"MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation","authors":"Nabil Ibtehaz, M Sohel Rahman","year":"2020","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional ad- versarial networks. In CVPR, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2019] Soo Ye Kim, Jihyong Oh, and Munchurl Kim. Deep SR-ITM: Joint learning of super-resolution and inverse tone-mapping for 4k UHD HDR applications. In ICCV, pages 3116-3125, 2019.","title":"Deep SR-ITM: Joint learning of super-resolution and inverse tone-mapping for 4k UHD HDR applications","authors":"Kim Soo Ye, Jihyong Oh, Munchurl Kim","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2020a] Jung Hee Kim, Siyeong Lee, and SukJu Kang. End-to-end differentiable learning to HDR image synthesis for multi-exposure images. AAAI, 2020.","title":"End-to-end differentiable learning to HDR image synthesis for multi-exposure images","authors":"Kim","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim et al., 2020b] Soo Ye Kim, Jihyong Oh, and Munchurl Kim. JSI-GAN: Gan-based joint super-resolution and inverse tone- mapping with pixel-wise task-specific filters for UHD HDR video. In AAAI, volume 34, pages 11287-11295, 2020.","title":"JSI-GAN: Gan-based joint super-resolution and inverse tonemapping with pixel-wise task-specific filters for UHD HDR video","authors":"Kim","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kovaleski and Oliveira, 2014] Rafael P Kovaleski and Manuel M Oliveira. High-quality reverse tone mapping for a wide range of exposures. In SIBGRAPI, pages 49-56. IEEE, 2014.","title":"High-quality reverse tone mapping for a wide range of exposures","authors":"P Rafael, Manuel M Kovaleski, Oliveira","year":"2014","venue":"SIBGRAPI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Landis, 2002] Hayden Landis. Production-ready global illumina- tion. In Siggraph, volume 5, pages 93-95, 2002.","title":"Production-ready global illumination","authors":"Hayden Landis","year":"2002","venue":"Siggraph","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2020] Yu-Lun Liu, WeiSheng Lai, YuSheng Chen, YiLung Kao, MingHsuan Yang, YungYu Chuang, and JiaBin Huang. Single-image HDR reconstruction by learning to reverse the camera pipeline. In CVPR, pages 1651-1660, 2020.","title":"Single-image HDR reconstruction by learning to reverse the camera pipeline","authors":"Yu-Lun Liu, Weisheng Lai, Yusheng Chen, Yilung Kao, Minghsuan Yang, Yungyu Chuang, Jiabin Huang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2021] Zhen Liu, Wenjie Lin, Xinpeng Li, Qing Rao, Ting Jiang, Mingyan Han, Haoqiang Fan, Jian Sun, and Shuaicheng Liu. ADNet: Attention-guided deformable convolu- tional network for high dynamic range imaging. In CVPR, pages 463-470, 2021.","title":"ADNet: Attention-guided deformable convolutional network for high dynamic range imaging","authors":"Zhen Liu, Wenjie Lin, Xinpeng Li, Qing Rao, Ting Jiang, Mingyan Han, Haoqiang Fan, Jian Sun, Shuaicheng Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mantiuk et al., 2011] Rafał Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. HDR-VDP-2: A calibrated vi- sual metric for visibility and quality predictions in all luminance conditions. ACM TOG, 30(4):1-14, 2011.","title":"HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions","authors":"Rafał Mantiuk, Joong Kil, Allan G Kim, Wolfgang Rempel, Heidrich","year":"2011","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pérez-Pellitero et al., 2021] Eduardo Pérez-Pellitero, Sibi Catley- Chandar, Ales Leonardis, and Radu Timofte. NTIRE 2021 chal- lenge on high dynamic range imaging: Dataset, methods and re- sults. In CVPR, pages 691-700, 2021.","title":"NTIRE 2021 challenge on high dynamic range imaging: Dataset, methods and results","authors":"Eduardo Pérez-Pellitero, Sibi Catley-Chandar, Ales Leonardis, Radu Timofte","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.","title":"U-Net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang and Yoon, 2021] Lin Wang and Kuk-Jin Yoon. Deep learn- ing for HDR imaging: State-of-the-art and future trends. arXiv preprint arXiv:2110.10394, 2021.","title":"Deep learning for HDR imaging: State-of-the-art and future trends","authors":"Lin Wang, Kuk-Jin Yoon","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Haonan Wang, Peng Cao, Jiaqi Wang, and Os- mar R Zaiane. UCTransNet: Rethinking the skip connections in U-Net from a channel-wise perspective with transformer. arXiv preprint arXiv:2109.04335, 2021.","title":"UCTransNet: Rethinking the skip connections in U-Net from a channel-wise perspective with transformer","authors":"Haonan Wang, Peng Cao, Jiaqi Wang, Osmar R Zaiane","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2021] Hanning Yu, Wentao Liu, Chengjiang Long, Bo Dong, Zou Qin, and Chunxia Xiao. Luminance attentive networks for HDR image and panorama reconstruction. arXiv preprint arXiv:2109.06688, 2021.","title":"Luminance attentive networks for HDR image and panorama reconstruction","authors":"Hanning Yu, Wentao Liu, Chengjiang Long, Bo Dong, Zou Qin, Chunxia Xiao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zeng et al., 2020] Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, and Lei Zhang. Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time. TPAMI, 2020.","title":"Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time","authors":"Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, Lei Zhang","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and Li, 2012] Lin Zhang and Hongyu Li. Sr-sim: A fast and high performance iqa index based on spectral residual. In ICIP, pages 1473-1476. IEEE, 2012.","title":"Sr-sim: A fast and high performance iqa index based on spectral residual","authors":"Lin Zhang, Hongyu Li","year":"2012","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2020] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. SG-one: Similarity guidance network for one- shot semantic segmentation. IEEE Trans Cybern, 50(9):3855- 3865, 2020.","title":"SG-one: Similarity guidance network for oneshot semantic segmentation","authors":"Xiaolin Zhang, Yunchao Wei, Yi Yang, Thomas S Huang","year":"2020","venue":"IEEE Trans Cybern","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pages 2223- 2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}