{"type":"ScholarlyArticle","title":"Learning Degradation Uncertainty for Unsupervised Real-world Image Super-resolution","authors":[{"name":"Qian Ning","slug":"qian-ning","given":"Qian","family":"Ning","affiliation":"Xidian University"},{"name":"Jingzhu Tang","slug":"jingzhu-tang","given":"Jingzhu","family":"Tang","affiliation":"Xidian University"},{"name":"Fangfang Wu","slug":"fangfang-wu","given":"Fangfang","family":"Wu","affiliation":"Xidian University"},{"name":"Weisheng Dong","slug":"weisheng-dong","given":"Weisheng","family":"Dong","affiliation":"Xidian University"},{"name":"Xin Li","slug":"xin-li","given":"Xin","family":"Li","affiliation":"West Virginia University"},{"name":"Guangming Shi","slug":"guangming-shi","given":"Guangming","family":"Shi","affiliation":"Xidian University"}],"abstract":"Acquiring degraded images with paired high-resolution (HR) images is often challenging, impeding the advance of image super-resolution in real-world applications. By generating realistic low-resolution (LR) images with degradation similar to that in real-world scenarios, simulated paired LR-HR data can be constructed for supervised training. However, most of the existing work ignores the degradation uncertainty of the generated realistic LR images, since only one LR image has been generated given an HR image. To address this weakness, we propose learning the degradation uncertainty of generated LR images and sampling multiple LR images from the learned LR image (mean) and degradation uncertainty (variance) and construct LR-HR pairs to train the super-resolution (SR) networks. Specifically, uncertainty can be learned by minimizing the proposed loss based on Kullback-Leibler (KL) divergence. Furthermore, the uncertainty in the feature domain is exploited by a novel perceptual loss; and we propose to calculate the adversarial loss from the gradient information in the SR stage for stable training performance and better visual quality. Experimental results on popular real-world datasets show that our proposed method has performed better than other unsupervised approaches.","doi":"10.24963/ijcai.2022/176","url":"https://ijcai.org/proceedings/2022/176","pdf":"https://ijcai.org/proceedings/2022/0176.pdf","markdown":"https://ijcai.org/proceedings/2022/176.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/176","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":176,"pages":{"first":1261,"last":1267},"keywords":["Uncertainty in AI: Uncertainty Representations","Computer Vision: Computational photography"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986534","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984677","length_sec":310,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bell-Kligler et al., 2019] Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan. In Advances in Neural Information Processing Systems, volume 32, 2019.","title":"Blind super-resolution kernel estimation using an internal-gan","authors":"Sefi Bell-Kligler, Assaf Shocher, Michal Irani","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2019] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single im- age super-resolution: A new benchmark and a new model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3086-3095, 2019.","title":"Toward real-world single image super-resolution: A new benchmark and a new model","authors":"Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, Lei Zhang","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2020] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.","title":"Data uncertainty learning in face recognition","authors":"Jie Chang, Zhonghao Lan, Changmao Cheng, Yichen Wei","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Chang Chen, Zhiwei Xiong, Xinmei Tian, Zheng-Jun Zha, and Feng Wu. Camera lens super- resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1652- 1660, 2019.","title":"Camera lens superresolution","authors":"Chang Chen, Zhiwei Xiong, Xinmei Tian, Zheng-Jun Zha, Feng Wu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2014] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional net- work for image super-resolution. In European Conference on Computer Vision, pages 184-199, 2014.","title":"Learning a deep convolutional network for image super-resolution","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fritsche et al., 2019] Manuel Fritsche, Shuhang Gu, and Radu Timofte. Frequency separation for real-world super- resolution. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019.","title":"Frequency separation for real-world superresolution","authors":"Shuhang Manuel Fritsche, Radu Gu, Timofte","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ignatov et al., 2017] Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. Dslr- quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 3277-3285, 2017.","title":"Dslrquality photos on mobile devices with deep convolutional networks","authors":"Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, Luc Van Gool","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ji et al., 2020] Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang. Real-world super- resolution via kernel estimation and noise injection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.","title":"Real-world superresolution via kernel estimation and noise injection","authors":"Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kendall and Gal, 2017] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In I.","title":"What uncertainties do we need in bayesian deep learning for computer vision? In I","authors":"Alex Kendall, Yarin Gal","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Guyon, U. V. Luxburg, S. Ben- gio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 30.","title":"Advances in Neural Information Processing Systems","authors":"U V Guyon, S Luxburg, H Bengio, R Wallach","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lim et al., 2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition Workshops (CVPRW), pages 1132-1140, 2017.","title":"Enhanced deep residual networks for single image super-resolution","authors":"Lim","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lugmayr et al., 2020] Andreas Lugmayr, Martin Danelljan, and Radu Timofte. Ntire 2020 challenge on real-world im- age super-resolution: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition Workshops, pages 494-495, 2020.","title":"Ntire 2020 challenge on real-world image super-resolution: Methods and results","authors":"Andreas Lugmayr, Martin Danelljan, Radu Timofte","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ning et al., 2021a] Qian Ning, Weisheng Dong, Xin Li, Jin- jian Wu, and Guangming Shi. Uncertainty-driven loss for single image super-resolution. Advances in Neural Infor- mation Processing Systems, 34, 2021.","title":"Uncertainty-driven loss for single image super-resolution","authors":"Ning","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ning et al., 2021b] Qian Ning, Weisheng Dong, Guang- ming Shi, Leida Li, and Xin Li. Accurate and lightweight image super-resolution with model-guided deep unfolding network. IEEE Journal of Selected Topics in Signal Pro- cessing, 15(2):240-252, 2021.","title":"Accurate and lightweight image super-resolution with model-guided deep unfolding network","authors":"Ning","year":"2021","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shocher et al., 2018] Assaf Shocher, Nadav Cohen, and Michal Irani. \"zero-shot\" super-resolution using deep in- ternal learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.","title":"zero-shot\" super-resolution using deep internal learning","authors":"Assaf Shocher, Nadav Cohen, Michal Irani","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Son et al., 2021] Sanghyun Son, Jaeha Kim, Wei-Sheng Lai, Ming-Hsuan Yang, and Kyoung Mu Lee. Toward real- world super-resolution via adaptive downsampling model- s. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.","title":"Toward realworld super-resolution via adaptive downsampling models","authors":"Sanghyun Son, Jaeha Kim, Wei-Sheng Lai, Ming-Hsuan Yang, Kyoung Mu, Lee","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018] Xintao Wang, Ke Yu, Shixiang Wu, Jin- jin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution genera- tive adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, 2018.","title":"Esrgan: Enhanced super-resolution generative adversarial networks","authors":"Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, Chen Change Loy","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV) workshops","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021a] Wei Wang, Haochen Zhang, Zehuan Yuan, and Changhu Wang. Unsupervised real-world super-resolution: A domain adaptation perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4318-4327, 2021.","title":"Unsupervised real-world super-resolution: A domain adaptation perspective","authors":"Wang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021b] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 1905-1914, 2021.","title":"Real-esrgan: Training real-world blind super-resolution with pure synthetic data","authors":"Wang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wei et al., 2021] Yunxuan Wei, Shuhang Gu, Yawei Li, Radu Timofte, Longcun Jin, and Hengjie Song. Unsu- pervised real-world image super resolution via domain- distance aware training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13385-13394, 2021.","title":"Unsupervised real-world image super resolution via domaindistance aware training","authors":"Wei","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wolf et al., 2021] Valentin Wolf, Andreas Lugmayr, Mar- tin Danelljan, Luc Van Gool, and Radu Timofte. Deflow: Learning complex image degradations from unpaired data with conditional flows. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.","title":"Deflow: Learning complex image degradations from unpaired data with conditional flows","authors":"Valentin Wolf, Andreas Lugmayr, Martin Danelljan, Luc Van Gool, Radu Timofte","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2018] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. In Proceedings of the European Conference on Computer Vision (ECCV), pages 286-301, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}