{"type":"ScholarlyArticle","title":"SS-BSN: Attentive Blind-Spot Network for Self-Supervised Denoising with Nonlocal Self-Similarity","authors":[{"name":"Young-Joo Han","slug":"young-joo-han","given":"Young-Joo","family":"Han","affiliation":"School of Computer Science , University of Seoul; Advanced Technology R&D Center , Vieworks"},{"name":"Ha-Jin Yu","slug":"ha-jin-yu","given":"Ha-Jin","family":"Yu","affiliation":"School of Computer Science , University of Seoul"}],"abstract":"Recently, numerous studies have been conducted on supervised learning-based image denoising methods. However, these methods rely on large-scale noisy-clean image pairs, which are difficult to obtain in practice. Denoising methods with self-supervised training that can be trained with only noisy images have been proposed to address the limitation. These methods are based on the convolutional neural network (CNN) and have shown promising performance. However, CNN-based methods do not consider using nonlocal self-similarities essential in the traditional method, which can cause performance limitations. This paper presents self-similarity attention (SS-Attention), a novel self-attention module that can capture nonlocal self-similarities to solve the problem. We focus on designing a lightweight self-attention module in a pixel-wise manner, which is nearly impossible to implement using the classic self-attention module due to the quadratically increasing complexity with spatial resolution. Furthermore, we integrate SS-Attention into the blind-spot network called self-similarity-based blind-spot network (SS-BSN). We conduct the experiments on real-world image denoising tasks. The proposed method quantitatively and qualitatively outperforms state-of-the-art methods in self-supervised denoising on the Smartphone Image Denoising Dataset (SIDD) and Darmstadt Noise Dataset (DND) benchmark datasets.","doi":"10.24963/ijcai.2023/89","url":"https://ijcai.org/proceedings/2023/89","pdf":"https://ijcai.org/proceedings/2023/0089.pdf","markdown":"https://ijcai.org/proceedings/2023/89.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/89","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":89,"pages":{"first":800,"last":809},"keywords":["Computer Vision: CV: Computational photography","Machine Learning: ML: Self-supervised Learning","Computer Vision: CV: Machine learning for vision"],"videos":[],"references":[{"position":1,"raw":"[SIDD Abdelhamed et al., 2018] DND [","title":null,"authors":"Sidd Abdelhamed","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Plotz and Roth, 2017] PSNR ↑ (dB) / SSIM ↑ PSNR ↑ (dB) / SSIM ↑ Non-learning Based BM3D [Dabov et al., 2007] 25.65 / 0.685 34.51 / 0.851 WNNM [Gu et al., 2014] 25.78 / 0.809 34.67 / 0.865 Supervised DnCNN [Zhang et al., 2017] 36.63 / 0.920 † 38.00 / 0.934 † DANet [Yue et al., 2020] 39.46 / 0.956 39.47 / 0.955 Supervised (Synthetic pairs) CBDNet [Guo et al., 2019] 33.28 / 0.868 38.05 / 0.942","title":"PSNR ↑ (dB) / SSIM ↑ PSNR ↑ (dB) / SSIM ↑ Non-learning Based BM3D","authors":"Roth Plotz","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Zhou et al. [Zhou et al., 2020] 34.02 / 0.898 † 38.40 / 0.945 Self-Supervised Noise2Void [Krull et al., 2019] 27.68 / 0.668 - Noise2Self [Batson and Royer, 2019] 29.59 / 0.808 - R2R [Pang et al., 2021] 34.78 / 0.898 - AP-BSN [Lee et al., 2022] 35.97 / 0.909 † 37.46 / 0.924 AP-BSNe [Lee et al., 2022] 37.05 / 0.934 38.09 / 0.937 Ours (Self-Supervised) SS-BSN 36.73 / 0.923 37.72 / 0.928 SS-BSNe 37.42 / 0.937 38.46 / 0.940","title":null,"authors":"Zhou","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Abdelhamed et al., 2018] Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.","title":"A high-quality denoising dataset for smartphone cameras","authors":"Abdelrahman Abdelhamed, Stephen Lin, Michael S Brown","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Batson and Royer, 2019] Joshua Batson and Loic Royer. Noise2Self: Blind denoising by self-supervision. In Ka- malika Chaudhuri and Ruslan Salakhutdinov, editors, Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 524-533. PMLR, 09-15 Jun 2019.","title":"Noise2Self: Blind denoising by self-supervision","authors":"Royer Batson, Joshua Batson, Loic Royer","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Buades et al., 2005] A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 2, pages 60-65 vol. 2, 2005.","title":"A non-local algorithm for image denoising","authors":"A Buades, B Coll, J.-M Morel","year":"2005","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12299-12310, June 2021.","title":"Pre-trained image processing transformer","authors":"Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, Wen Gao","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dabov et al., 2007] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):2080-2095, 2007.","title":"Image denoising by sparse 3-d transform-domain collaborative filtering","authors":"Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, Karen Egiazarian","year":"2007","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2014] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimiza- tion with application to image denoising. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 2862-2869, 2014.","title":"Weighted nuclear norm minimization with application to image denoising","authors":"Gu","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guo et al., 2019] Shi Guo, Zifei Yan, Kai Zhang, Wang- meng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.","title":"Toward convolutional blind denoising of real photographs","authors":"Zifei Shi Guo, Kai Yan, Wangmeng Zhang, Lei Zuo, Zhang","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krull et al., 2019] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void -learning denoising from single noisy images. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2124-2132, 2019.","title":"Noise2void -learning denoising from single noisy images","authors":"Alexander Krull, Tim-Oliver Buchholz, Florian Jug","year":"2019","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2022] Wooseok Lee, Sanghyun Son, and Ky- oung Mu Lee. Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot net- work. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17725-17734, June 2022.","title":"Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network","authors":"Lee","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lehtinen et al., 2018] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2Noise: Learning image restoration without clean data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pages 2965-2974. PMLR, 10- 15 Jul 2018.","title":"Noise2Noise: Learning image restoration without clean data","authors":"Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, Timo Aila","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision (ICCV) Workshops, pages 1833-1844, Octo- ber 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Niu et al., 2020] Chuang Niu, Mengzhou Li, Fenglei Fan, Weiwen Wu, Xiaodong Guo, Qing Lyu, and Ge Wang. Suppression of Correlated Noise with Similarity-based Unsupervised Deep Learning. arXiv e-prints, page arXiv:2011.03384, November 2020.","title":"Suppression of Correlated Noise with Similarity-based Unsupervised Deep Learning","authors":"Chuang Niu, Mengzhou Li, Fenglei Fan, Weiwen Wu, Xiaodong Guo, Qing Lyu, Ge Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pang et al., 2021] Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji. Recorrupted-to-recorrupted: Unsupervised deep learning for image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2043-2052, June 2021.","title":"Recorrupted-to-recorrupted: Unsupervised deep learning for image denoising","authors":"Pang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peyré et al., 2011] Gabriel Peyré, Sébastien Bougleux, and Laurent D. Cohen. Non-local Regularization of Inverse Problems. Inverse Problems and Imaging , 5(2):511-530, 2011.","title":"Non-local Regularization of Inverse Problems","authors":"Gabriel Peyré, Sébastien Bougleux, Laurent D Cohen","year":"2011","venue":"Inverse Problems and Imaging","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Plotz and Roth, 2017] Tobias Plotz and Stefan Roth. Bench- marking denoising algorithms with real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.","title":"Benchmarking denoising algorithms with real photographs","authors":"Roth ; Plotz, Tobias Plotz, Stefan Roth","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Quan et al., 2020] Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2self with dropout: Learning self- supervised denoising from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2020.","title":"Self2self with dropout: Learning selfsupervised denoising from single image","authors":"Yuhui Quan, Mingqin Chen, Tongyao Pang, Hui Ji","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tu et al., 2022] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Gio- vanni Maria Farinella, and Tal Hassner, editors, Com- puter Vision -ECCV 2022, pages 459-479, Cham, 2022. Springer Nature Switzerland.","title":"Maxvit: Multi-axis vision transformer","authors":"Tu","year":"2022","venue":"Computer Vision -ECCV 2022","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 30.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, Illia Polosukhin","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2020] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep im- age denoising. In Computer Vision -ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, page 352-368, Berlin, Heidelberg, 2020. Springer-Verlag.","title":"Unpaired learning of deep image denoising","authors":"Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, Wangmeng Zuo","year":"2020","venue":"Computer Vision -ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2015] Jun Xu, Lei Zhang, Wangmeng Zuo, David Zhang, and Xiangchu Feng. Patch group based nonlocal self-similarity prior learning for image denoising. In Pro- ceedings of the IEEE International Conference on Com- puter Vision (ICCV), December 2015.","title":"Patch group based nonlocal self-similarity prior learning for image denoising","authors":"Jun Xu, Lei Zhang, Wangmeng Zuo, David Zhang, Xiangchu Feng","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yue et al., 2020] Zongsheng Yue, Qian Zhao, Lei Zhang, and Deyu Meng. Dual adversarial network: Toward real- world noise removal and noise generation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision -ECCV 2020, pages 41- 58, Cham, 2020. Springer International Publishing.","title":"Dual adversarial network: Toward realworld noise removal and noise generation","authors":"Zongsheng Yue, Qian Zhao, Lei Zhang, Deyu Meng","year":"2020","venue":"Computer Vision -ECCV 2020","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zamir et al., 2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5728-5739, June 2022.","title":"Restormer: Efficient transformer for high-resolution image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Yang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2017] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denois- ing. IEEE Transactions on Image Processing, 26(7):3142- 3155, 2017.","title":"Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising","authors":"Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, Lei Zhang","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2020] Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, and Thomas Huang. When awgn-based denoiser meets real noises. Proceed- ings of the AAAI Conference on Artificial Intelligence, 34(07):13074-13081, Apr. 2020.","title":"When awgn-based denoiser meets real noises","authors":"Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, Thomas Huang","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}