{"type":"ScholarlyArticle","title":"KPN-MFI: A Kernel Prediction Network with Multi-frame Interaction for Video Inverse Tone Mapping","authors":[{"name":"Gaofeng Cao","slug":"gaofeng-cao","given":"Gaofeng","family":"Cao","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"},{"name":"Fei Zhou","slug":"fei-zhou","given":"Fei","family":"Zhou","affiliation":"College of Electronic and Information Engineering , Shenzhen University; Peng Cheng Laboratory; Guangdong Key Laboratory of Intelligent Information Processing; Shenzhen Key Laboratory of Digital Creative Technology; Shenzhen Institute for Artificial Intelligence and Robotics for Society (AIRS)"},{"name":"Han Yan","slug":"han-yan","given":"Han","family":"Yan","affiliation":"Department of Computer Science , Harbin Institute of Technology"},{"name":"Anjie Wang","slug":"anjie-wang","given":"Anjie","family":"Wang","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"},{"name":"Leidong Fan","slug":"leidong-fan","given":"Leidong","family":"Fan","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"}],"abstract":"Up to now, the image-based inverse tone mapping (iTM) models have been widely investigated, while there is little research on video-based iTM methods. It would be interesting to make use of these existing image-based models in the video iTM task. However, directly transferring the imagebased iTM models to video data without modeling spatial-temporal information remains nontrivial and challenging. Considering both the intra-frame quality and the inter-frame consistency of a video, this article presents a new video iTM method based on a kernel prediction network (KPN), which takes advantage of multi-frame interaction (MFI) module to capture temporal-spatial information for video data. Specifically, a basic encoder-decoder KPN, essentially designed for image iTM, is trained to guarantee the mapping quality within each frame. More importantly, the MFI module is incorporated to capture temporal-spatial context information and preserve the inter-frame consistency by exploiting the correction between adjacent frames. Notably, we can readily extend any existing image iTM models to video iTM ones by involving the proposed MFI module. Furthermore, we propose an inter-frame brightness consistency loss function based on the Gaussian pyramid to reduce the video temporal inconsistency. Extensive experiments demonstrate that our model outperforms state-ofthe-art image and video-based methods. The code\r\nis available at https://github.com/caogaofeng/KPNMFI.","doi":"10.24963/ijcai.2022/113","url":"https://ijcai.org/proceedings/2022/113","pdf":"https://ijcai.org/proceedings/2022/0113.pdf","markdown":"https://ijcai.org/proceedings/2022/113.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/113","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":113,"pages":{"first":806,"last":812},"keywords":["Computer Vision: Computational photography","Computer Vision: Applications","Machine Learning: Convolutional Networks","Computer Vision: Machine Learning for Vision"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986992","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985135","length_sec":611,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cao et al., 2021] Gaofeng Cao, Fei Zhou, Kanglin Liu, and Liu Bozhi. A brightness-adaptive kernel predic- tion network for inverse tone mapping. Neurocomputing, 464(13):1-14, 2021.","title":"A brightness-adaptive kernel prediction network for inverse tone mapping","authors":"Gaofeng Cao, Fei Zhou, Kanglin Liu, Liu Bozhi","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021a] Weitao Chen, Zhibin Wang, and Hao Li. Get better 1 pixel pck: Ladder scales correspondence flow networks for remote sensing image matching in higher res- olution. In IEEE/CVF CVPR, pages 742-751, 2021.","title":"Get better 1 pixel pck: Ladder scales correspondence flow networks for remote sensing image matching in higher resolution","authors":"Chen","year":"2021","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021b] Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, and Chao Dong. Hdrunet: Single im- age hdr reconstruction with denoising and dequantization. In IEEE/CVF CVPRW, pages 354-363, 2021.","title":"Hdrunet: Single image hdr reconstruction with denoising and dequantization","authors":"Chen","year":"2021","venue":"IEEE/CVF CVPRW","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021c] Xiangyu Chen, Zhengwen Zhang, Jimmy S Ren, Lynhoo Tian, Yu Qiao, and Chao Dong. A new journey from sdrtv to hdrtv. In IEEE/CVF ICCV, pages 4500-4509, 2021.","title":"A new journey from sdrtv to hdrtv","authors":"Chen","year":"2021","venue":"IEEE/CVF ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hui and Loy, 2020] Tak-Wai Hui and Chen Change Loy. Liteflownet3: Resolving correspondence ambiguity for more accurate optical flow estimation. In ECCV, pages 169-184. Springer, 2020.","title":"Liteflownet3: Resolving correspondence ambiguity for more accurate optical flow estimation","authors":"Loy ; ] Tak-Wai Hui, Chen Change Hui, Loy","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huo et al., 2014] Yongqing Huo, Fan Yang, Le Dong, and Vincent Brost. Physiological inverse tone mapping based on retina response. The Visual Computer, 30(5):507-517, 2014.","title":"Physiological inverse tone mapping based on retina response","authors":"Yongqing Huo, Fan Yang, Le Dong, Vincent Brost","year":"2014","venue":"The Visual Computer","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[ITU-R, 2011] ITU-R. Reference electro-optical transfer function for flat panel displays used in hdtv studio pro- duction. ITU-R Rec, BT.1886, 2011.","title":"ITU-R. Reference electro-optical transfer function for flat panel displays used","authors":"Itu-R","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[ITU-R, 2014] ITU-R. High dynamic range electro-optical transfer function of mastering reference displays. SMPTE ST2084:2014, 2014.","title":"ITU-R. High dynamic range electro-optical transfer function of mastering reference displays","authors":"Itu-R","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[ITU-R, 2015a] ITU-R. Parameter values for the hdtv stan- dards for production and international programme ex- change. ITU-R Rec, BT.709-6, 2015.","title":"ITU-R. Parameter values for the hdtv standards for production and international programme exchange","authors":"Itu-R","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[ITU-R, 2015b] ITU-R. Parameter values for ultra-high def- inition television systems for production and international programme exchange. ITU-R Rec, BT.2020-2, 2015.","title":"ITU-R. Parameter values for ultra-high definition television systems for production and international programme exchange. ITU-R Rec","authors":"Itu-R","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[ITU-R, 2019] ITU-R. Objective metric for the assessment of the potential visibility of colour differences in televi- sion. ITU-R Rec, BT.2124-0, 2019.","title":"ITU-R. Objective metric for the assessment of the potential visibility of colour differences in television","authors":"Itu-R","year":"2019","venue":"ITU-R Rec","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang and Zheng, 2019] Haiyang Jiang and Yinqiang Zheng. Learning to see moving objects in the dark. In IEEE/CVF CVPR, pages 7324-7333, 2019.","title":"Learning to see moving objects in the dark","authors":"Zheng Jiang, Haiyang Jiang, Yinqiang Zheng","year":"2019","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2019] Soo Ye Kim, Jihyong Oh, and Munchurl Kim. Deep sr-itm: Joint learning of super-resolution and inverse tone-mapping for 4k uhd hdr applications. In IEEE/CVF ICCV, pages 3116-3125, 2019.","title":"Deep sr-itm: Joint learning of super-resolution and inverse tone-mapping for 4k uhd hdr applications","authors":"Kim Soo Ye, Jihyong Oh, Munchurl Kim","year":"2019","venue":"IEEE/CVF ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2020] Soo Ye Kim, Jihyong Oh, and Munchurl Kim. Jsi-gan: Gan-based joint super-resolution and in- verse tone-mapping with pixel-wise task-specific filters for uhd hdr video. In AAAI, volume 34, pages 11287-11295, 2020.","title":"Jsi-gan: Gan-based joint super-resolution and inverse tone-mapping with pixel-wise task-specific filters for uhd hdr video","authors":"Kim Soo Ye, Jihyong Oh, Munchurl Kim","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lai et al., 2018] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In ECCV, pages 170-185, 2018.","title":"Learning blind video temporal consistency","authors":"Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, Ming-Hsuan Yang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Edwin H. Land and John J. Mc- Cann. Lightness and retinex theory. J. Opt. Soc. Am., 61(1):1-11, 1971.","title":"Lightness and retinex theory","authors":"Edwin H Land, John J Mc-Cann","year":"1971","venue":"J. Opt. Soc. Am","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2021] Huangxing Lin, Yihong Zhuang, Yue Huang, Xinghao Ding, Xiaoqing Liu, and Yizhou Yu. Noise2grad: Extract image noise to denoise. In IJCAI, pages 830-836, 2021.","title":"Noise2grad: Extract image noise to denoise","authors":"Huangxing Lin, Yihong Zhuang, Yue Huang, Xinghao Ding, Xiaoqing Liu, Yizhou Yu","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/115"},{"position":18,"raw":"[Liu et al., 2020] Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. Single-image hdr reconstruction by learning to reverse the camera pipeline. In IEEE/CVF CVPR, pages 1651-1660, 2020.","title":"Single-image hdr reconstruction by learning to reverse the camera pipeline","authors":"Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, Jia-Bin Huang","year":"2020","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mantiuk et al., 2011] Rafał Mantiuk, Kil Joong Kim, Al- lan G. Rempel, and Wolfgang Heidrich. Hdr-vdp-2: A calibrated visual metric for visibility and quality predic- tions in all luminance conditions. ACM TOG, 30(4):1-14, 2011.","title":"Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions","authors":"Rafał Mantiuk, Joong Kil, Allan G Kim, Wolfgang Rempel, Heidrich","year":"2011","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Santos et al., 2020] Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. Single image hdr recon- struction using a cnn with masked features and perceptual loss. ACM TOG, 39(4):1-10, 2020.","title":"Single image hdr reconstruction using a cnn with masked features and perceptual loss","authors":"Santos","year":"2020","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In IEEE/CVF CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2019a] Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen. Deep exemplar-based video colorization. In IEEE/CVF CVPR, pages 8052-8061, 2019.","title":"Deep exemplar-based video colorization","authors":"Zhang","year":"2019","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2019b] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, volume 97, pages 7354- 7363, 2019.","title":"Self-attention generative adversarial networks","authors":"Zhang","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2021] Guoqing Zhang, Yuhao Chen, Weisi Lin, Arun Chandran, and Xuan Jing. Low resolution in- formation also matters: Learning multi-resolution repre- sentations for person re-identification. In IJCAI, pages 1295-1301, 2021.","title":"Low resolution information also matters: Learning multi-resolution representations for person re-identification","authors":"Guoqing Zhang, Yuhao Chen, Weisi Lin, Arun Chandran, Xuan Jing","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/179"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}