{"type":"ScholarlyArticle","title":"QFormer: An Efficient Quaternion Transformer for Image Denoising","authors":[{"name":"Bo Jiang","slug":"bo-jiang","given":"Bo","family":"Jiang","affiliation":"College of Mechanical and Electronic Engineering , Northwest A&F University , China"},{"name":"Yao Lu","slug":"yao-lu","given":"Yao","family":"Lu","affiliation":"Department of Computer Science , Harbin Institute of Technology at Shenzhen , China"},{"name":"Guangming Lu","slug":"guangming-lu","given":"Guangming","family":"Lu","affiliation":"Department of Computer Science , Harbin Institute of Technology at Shenzhen , China"},{"name":"Bob Zhang","slug":"bob-zhang","given":"Bob","family":"Zhang","affiliation":"Department of Computer and Information Science , University of Macau , China jiangbo"}],"abstract":"Since Deep Convolutional Neural Networks (DCNNs) and Vision Transformer perform well in learning generalizable image priors from large-scale data, these models have been widely used in image denoising tasks. However, vanilla DCNNs and Transformer suffer from two problems. First, the vanilla DCNNs and Transformer only accumulate the output along the channel axis, ignoring the internal relationship among channels. This results in the severely inadequate color structure representation retrieved from color images. Secondly, the DCNNs or Transformer-based image denoising models usually have a large number of parameters, high computational complexity, and slow inference speed. To resolve these issues, this paper proposes a highly-efficient Quaternion Transformer (QFormer) for image denoising. Specifically, the proposed Quaternion Transformer Block (QTB) simplifies the typical Transformer from a multi-branch structure to an elaborately sequential structure mainly with quaternion transformations, to alternately capture both long-range dependencies and local contextual features with color structure information. Furthermore, the proposed QTB can also avoid considerable element-wise multiplications of computing the self-attention matrices. Thus, our QTB can significantly reduce the computational complexity and its sequential structure can further improve the practical inference speed. Comprehensive experiments demonstrate that the proposed QFormer produces state-of-the-art results in both denoising performance and efficiency. We hope that our work will encourage further research to explore the Quaternion Transformer architecture for image denoising tasks.","doi":"10.24963/ijcai.2024/468","url":"https://ijcai.org/proceedings/2024/468","pdf":"https://ijcai.org/proceedings/2024/0468.pdf","markdown":"https://ijcai.org/proceedings/2024/468.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/468","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":468,"pages":{"first":4237,"last":4245},"keywords":["Machine Learning: ML: Knowledge-aided learning"],"videos":[],"references":[{"position":1,"raw":"[Anwar and Barnes, 2019] Saeed Anwar and Nick Barnes. Real image denoising with feature attention. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3155-3164, 2019.","title":"Real image denoising with feature attention","authors":"Saeed Anwar, Nick Barnes","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Yongyong Chen, Xiaolin Xiao, and Yi- cong Zhou. Low-rank quaternion approximation for color image processing. IEEE Transactions on Image Process- ing, 29:1426-1439, 2020.","title":"Low-rank quaternion approximation for color image processing","authors":"Yongyong Chen, Xiaolin Xiao, Yicong Zhou","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ell and Sangwine, 2007] Todd A Ell and Stephen J Sang- wine. Quaternion involutions and anti-involutions. Com- puters & Mathematics with Applications, 53(1):137-143, 2007.","title":"Quaternion involutions and anti-involutions","authors":"A Todd, Stephen J Ell, Sangwine","year":"2007","venue":"Computers & Mathematics with Applications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2019] Shi Guo, Zifei Yan, K. Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denois- ing of real photographs. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1712-1722, 2019.","title":"Toward convolutional blind denoising of real photographs","authors":"Shi Guo, K Zifei Yan, Wangmeng Zhang, Lei Zuo, Zhang","year":"2019","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hamilton, 2022] William Rowan Hamilton. Elements of quaternions. BoD-Books on Demand, 2022.","title":"Elements of quaternions","authors":"William Rowan, Hamilton","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.","title":"Gaussian error linear units","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2022] Bo Jiang, Yao Lu, Jiahuan Wang, Guang- ming Lu, and David Zhang. Deep image denoising with adaptive priors. IEEE Transactions on Circuits and Sys- tems for Video Technology, 2022.","title":"Deep image denoising with adaptive priors","authors":"Bo Jiang, Yao Lu, Jiahuan Wang, Guangming Lu, David Zhang","year":"2022","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Joshi et al., 2005] Manjunath V Joshi, Subhasis Chaudhuri, and Rajkiran Panuganti. A learning-based method for im- age super-resolution from zoomed observations. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 35(3):527-537, 2005.","title":"A learning-based method for image super-resolution from zoomed observations","authors":"V Manjunath, Subhasis Joshi, Rajkiran Chaudhuri, Panuganti","year":"2005","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2020] Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and Nam Ik Cho. Transfer learning from synthetic to real-noise denoising with adaptive instance normaliza- tion. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3479-3489, 2020.","title":"Transfer learning from synthetic to real-noise denoising with adaptive instance normalization","authors":"Kim","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kolesnikov et al., 2021] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar et al., 2021] Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. Colorization transformer. arXiv preprint arXiv:2102.04432, 2021.","title":"Colorization transformer","authors":"Manoj Kumar, Dirk Weissenborn, Nal Kalchbrenner","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833-1844, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2021] Ruijun Ma, Bob Zhang, Yicong Zhou, Zhengmin Li, and Fangyuan Lei. Pid controller-guided at- tention neural network learning for fast and effective real photographs denoising. IEEE transactions on neural net- works and learning systems, PP, 2021.","title":"Pid controller-guided attention neural network learning for fast and effective real photographs denoising","authors":"Ma","year":"2021","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2022] Rui Ma, Shuyi Li, Bob Zhang, and Zheng- min Li. Towards fast and robust real image denoising with attentive neural network and pid controller. IEEE Trans- actions on Multimedia, pages 2366-2377, 2022.","title":"Towards fast and robust real image denoising with attentive neural network and pid controller","authors":"Rui Ma, Shuyi Li, Bob Zhang, Zhengmin Li","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Moxey et al., 2003] C Eddie Moxey, Stephen J Sangwine, and Todd A Ell. Hypercomplex correlation techniques for vector images. IEEE Transactions on Signal Processing, 51(7):1941-1953, 2003.","title":"Hypercomplex correlation techniques for vector images","authors":"Eddie Moxey, Stephen J Sangwine, Todd A Ell","year":"2003","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Murray et al., 1994] Richard M. Murray, S. Shankar Sastry, and Li Ze-xiang. A mathematical introduction to robotic manipulation. 1994. [Parcollet et al., 2019] Titouan Parcollet, Mohamed Morchid, and Georges Linarès. Quaternion convolutional neural networks for heterogeneous image processing. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8514-8518. IEEE, 2019.","title":"Titouan Parcollet, Mohamed Morchid, and Georges Linarès. Quaternion convolutional neural networks for heterogeneous image processing","authors":"Richard M Murray, S Shankar Sastry, Li Ze-Xiang ; Parcollet","year":"1994","venue":"ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2021] Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. Addersr: Towards energy efficient image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15648-15657, 2021.","title":"Addersr: Towards energy efficient image super-resolution","authors":"Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, Dacheng Tao","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021a] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2021b] Zhendong Wang, Xiaodong Cun, Jian- min Bao, and Jianzhuang Liu. Uformer: A general u- shaped transformer for image restoration. arXiv preprint arXiv:2106.03106, 2021.","title":"Uformer: A general ushaped transformer for image restoration","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2021a] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 22-31, 2021.","title":"Cvt: Introducing convolutions to vision transformers","authors":"Wu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2021b] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and im- proving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10033-10041, 2021.","title":"Rethinking and improving relative position encoding for vision transformer","authors":"Wu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2015] Yi Xu, Licheng Yu, Hongteng Xu, Hao Zhang, and Truong Nguyen. Vector sparse represen- tation of color image using quaternion matrix analy- sis. IEEE Transactions on image processing, 24(4):1315- 1329, 2015.","title":"Vector sparse representation of color image using quaternion matrix analysis","authors":"Yi Xu, Licheng Yu, Hongteng Xu, Hao Zhang, Truong Nguyen","year":"2015","venue":"IEEE Transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2017] Jun Xu, Lei Zhang, David Zhang, and Xi- angchu Feng. Multi-channel weighted nuclear norm min- imization for real color image denoising. In Proceedings of the IEEE international conference on computer vision, pages 1096-1104, 2017.","title":"Multi-channel weighted nuclear norm minimization for real color image denoising","authors":"Jun Xu, Lei Zhang, David Zhang, Xiangchu Feng","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2018] Jun Xu, Lei Zhang, and David Dian Zhang. A trilateral weighted sparse coding scheme for real-world image denoising. ArXiv, abs/1807.04364, 2018.","title":"A trilateral weighted sparse coding scheme for real-world image denoising","authors":"Jun Xu, Lei Zhang, David Dian Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yin et al., 2012] Ming Yin, Wei Liu, Jun Shui, and Jiang- min Wu. Quaternion wavelet analysis and application in image denoising. Mathematical Problems in Engineering, 2012, 2012.","title":"Quaternion wavelet analysis and application in image denoising","authors":"Ming Yin, Wei Liu, Jun Shui, Jiangmin Wu","year":"2012","venue":"Mathematical Problems in Engineering","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2021] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. arXiv preprint arXiv:2111.11418, 2021.","title":"Metaformer is actually what you need for vision","authors":"Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yue et al., 2019] Zongsheng Yue, Hongwei Yong, Qian Zhao, Lei Zhang, and Deyu Meng. Variational denoising network: Toward blind noise modeling and removal. In NeurIPS, 2019.","title":"Variational denoising network: Toward blind noise modeling and removal","authors":"Zongsheng Yue, Hongwei Yong, Qian Zhao, Lei Zhang, Deyu Meng","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zamir et al., 2020] Syed Waqas Zamir, Aditya Arora, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning en- riched features for real image restoration and enhance- ment. ArXiv, abs/2003.06792, 2020.","title":"Learning enriched features for real image restoration and enhancement","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Hameed Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Ling Yang, Shao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zamir et al., 2021] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. arXiv preprint arXiv:2111.09881, 2021.","title":"Restormer: Efficient transformer for high-resolution image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zeng et al., 2016] Rui Zeng, Jiasong Wu, Zhuhong Shao, Yang Chen, Beijing Chen, Lotfi Senhadji, and Huazhong Shu. Color image classification via quaternion principal component analysis network. Neurocomputing, 216:416- 428, 2016.","title":"Color image classification via quaternion principal component analysis network","authors":"Rui Zeng, Jiasong Wu, Zhuhong Shao, Yang Chen, Beijing Chen, Lotfi Senhadji, Huazhong Shu","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2017] K. Zhang, W. Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Resid- ual learning of deep cnn for image denoising. IEEE Trans- actions on Image Processing, 26:3142-3155, 2017.","title":"Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising","authors":"K Zhang, W Zuo, Yunjin Chen, Deyu Meng, Lei Zhang","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2018] K. Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing, 27:4608-4622, 2018.","title":"Ffdnet: Toward a fast and flexible solution for cnn-based image denoising","authors":"K Zhang, Wangmeng Zuo, Lei Zhang","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2018] Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural net- works. In Proceedings of the European Conference on Computer Vision (ECCV), pages 631-647, 2018.","title":"Quaternion convolutional neural networks","authors":"Xuanyu Zhu, Yi Xu, Hongteng Xu, Changjian Chen","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zou et al., 2016] Cuiming Zou, Kit Ian Kou, and Yulong Wang. Quaternion collaborative and sparse representation with application to color face recognition. IEEE Transac- tions on image processing, 25(7):3287-3302, 2016.","title":"Quaternion collaborative and sparse representation with application to color face recognition","authors":"Zou","year":"2016","venue":"IEEE Transactions on image processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}