{"type":"ScholarlyArticle","title":"Residual Contrastive Learning for Image Reconstruction: Learning Transferable Representations from Noisy Images","authors":[{"name":"Nanqing Dong","slug":"nanqing-dong","given":"Nanqing","family":"Dong","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Matteo Maggioni","slug":"matteo-maggioni","given":"Matteo","family":"Maggioni","affiliation":"Huawei Noah's Ark Lab"},{"name":"Yongxin Yang","slug":"yongxin-yang","given":"Yongxin","family":"Yang","affiliation":"Huawei Noah's Ark Lab"},{"name":"Eduardo Pérez-Pellitero","slug":"eduardo-perez-pellitero","given":"Eduardo","family":"Pérez-Pellitero","affiliation":"Huawei Noah's Ark Lab"},{"name":"Ales Leonardis","slug":"ales-leonardis","given":"Ales","family":"Leonardis","affiliation":"Huawei Noah's Ark Lab"},{"name":"Steven McDonagh","slug":"steven-mcdonagh","given":"Steven","family":"McDonagh","affiliation":"Huawei Noah's Ark Lab"}],"abstract":"This paper is concerned with contrastive learning (CL) for low-level image restoration and enhancement tasks. We propose a new label-efficient learning paradigm based on residuals, residual contrastive learning (RCL), and derive an unsupervised visual representation learning framework, suitable for low-level vision tasks with noisy inputs. While supervised image reconstruction aims to minimize residual terms directly, RCL alternatively builds a connection between residuals and CL by defining a novel instance discrimination pretext task, using residuals as the discriminative feature. Our formulation mitigates the severe task misalignment between instance discrimination pretext tasks and downstream image reconstruction tasks, present in existing CL frameworks. Experimentally, we find that RCL can learn robust and transferable representations that improve the performance of various downstream tasks, such as denoising and super resolution, in comparison with recent self-supervised methods designed specifically for noisy inputs. Additionally, our unsupervised pre-training can significantly reduce annotation costs whilst maintaining performance competitive with fully-supervised image reconstruction.","doi":"10.24963/ijcai.2022/406","url":"https://ijcai.org/proceedings/2022/406","pdf":"https://ijcai.org/proceedings/2022/0406.pdf","markdown":"https://ijcai.org/proceedings/2022/406.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/406","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":406,"pages":{"first":2930,"last":2936},"keywords":["Machine Learning: Self-supervised Learning","Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Machine Learning: Multi-task and Transfer Learning","Machine Learning: Unsupervised Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986780","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984923","length_sec":355,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Batson and Royer, 2019] Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In ICML, pages 524-533. PMLR, 2019.","title":"Noise2self: Blind denoising by self-supervision","authors":"Royer Batson, Joshua Batson, Loic Royer","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chuang et al., 2020] Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In NIPS, volume 33, pages 8765-8775, 2020.","title":"Debiased contrastive learning","authors":"Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, Stefanie Jegelka","year":"2020","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2021] Nanqing Dong, Michael Kampffmeyer, and Irina Voiculescu. Self-supervised multi-task represen- tation learning for sequential medical images. In ECML, pages 779-794. Springer, 2021.","title":"Self-supervised multi-task representation learning for sequential medical images","authors":"Dong","year":"2021","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ehret et al., 2019] Thibaud Ehret, Axel Davy, Pablo Arias, and Gabriele Facciolo. Joint demosaicking and denoising by fine-tuning of bursts of raw images. In ICCV, pages 8868-8877, 2019.","title":"Joint demosaicking and denoising by fine-tuning of bursts of raw images","authors":"Thibaud Ehret, Axel Davy, Pablo Arias, Gabriele Facciolo","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ericsson et al., 2021] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In CVPR, pages 5414-5423, 2021.","title":"How well do self-supervised models transfer? In CVPR","authors":"Ericsson","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisser- man. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303-338, 2010.","title":"The pascal visual object classes (VOC) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gharbi et al., 2016] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. Deep joint demosaicking and denoising. ACM TOG, 35(6):1-12, 2016.","title":"Deep joint demosaicking and denoising","authors":"Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, Frédo Durand","year":"2016","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hasinoff et al., 2010] Samuel W Hasinoff, Frédo Durand, and William T Freeman. Noise-optimal capture for high dynamic range photography. In CVPR, pages 553-560. IEEE, 2010.","title":"Noise-optimal capture for high dynamic range photography","authors":"Frédo Samuel W Hasinoff, William T Durand, Freeman","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. [","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lehtinen et al., 2018] Jaakko Lehtinen, Jacob Munkberg, and et al. Noise2noise: Learning image restoration with- out clean data. In ICML, pages 2965-2974. PMLR, 2018.","title":"Noise2noise: Learning image restoration without clean data","authors":"Lehtinen","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018] Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang. Multi-scale residual network for image super-resolution. In ECCV, pages 517-532, 2018.","title":"Multi-scale residual network for image super-resolution","authors":"Li","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In ICLR, 2021.","title":"Prototypical contrastive learning of unsupervised representations","authors":"Li","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2020] Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without ima- genet. arXiv preprint arXiv:2011.13677, 2020.","title":"Self-emd: Self-supervised object detection without imagenet","authors":"Songtao Liu, Zeming Li, Jian Sun","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[O. Pinheiro et al., 2020] Pedro O O. Pinheiro, Amjad Almahairi, and et al. Unsupervised learning of dense vi- sual representations. In NIPS, volume 33, pages 4489- 4500.","title":"Unsupervised learning of dense visual representations","authors":"O O Pedro, Amjad Pinheiro, Almahairi","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Oord","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Isola, 2020] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, pages 9929-9939. PMLR, 2020.","title":"Understanding contrastive representation learning through alignment and uniformity on the hypersphere","authors":"Wang, Isola ;, Tongzhou Wang, Phillip Isola","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021a] Xinlong Wang, Rufeng Zhang, Chun- hua Shen, Tao Kong, and Lei Li. Dense contrastive learn- ing for self-supervised visual pre-training. In CVPR, pages 3024-3033, 2021.","title":"Dense contrastive learning for self-supervised visual pre-training","authors":"Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2021b] Yanbo Wang, Shaohui Lin, and et al. Towards compact single image super-resolution via con- trastive self-distillation. In IJCAI, pages 1122-1128, 2021.","title":"Towards compact single image super-resolution via contrastive self-distillation","authors":"Wang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/155"},{"position":26,"raw":"[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, pages 3733- 3742, 2018.","title":"Unsupervised feature learning via nonparametric instance discrimination","authors":"Zhirong Wu, Yuanjun Xiong, Stella X Yu, Dahua Lin","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2021] Haiyan Wu, Yanyun Qu, and et al. Con- trastive learning for compact single image dehazing. In CVPR, pages 10551-10560, 2021.","title":"Contrastive learning for compact single image dehazing","authors":"Haiyan Wu, Yanyun Qu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2021] Zhenda Xie, Yutong Lin, and et al. Prop- agate yourself: Exploring pixel-level consistency for un- supervised visual representation learning. In CVPR, pages 16684-16693, 2021.","title":"Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning","authors":"Xie","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zamir et al., 2018] Amir R Zamir, Alexander Sax, and et al. Taskonomy: Disentangling task transfer learning. In CVPR, pages 3712-3722, 2018.","title":"Taskonomy: Disentangling task transfer learning","authors":"Alexander Amir R Zamir, Sax","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":"/proceedings/2019/871"},{"position":30,"raw":"[Zhang et al., 2016] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, pages 649-666. Springer, 2016.","title":"Colorful image colorization","authors":"Richard Zhang, Phillip Isola, Alexei A Efros","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2017] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian de- noiser: Residual learning of deep cnn for image denoising. IEEE TIP, 26(7):3142-3155, 2017.","title":"Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising","authors":"Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, Lei Zhang","year":"2017","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}