{"type":"ScholarlyArticle","title":"FreqFormer: Frequency-aware Transformer for Lightweight Image Super-resolution","authors":[{"name":"Tao Dai","slug":"tao-dai","given":"Tao","family":"Dai","affiliation":"College of Computer Science and Software Engineering , Shenzhen University; National Engineering Laboratory for Big Data System Computing Technology , Shenzhen University"},{"name":"Jianping Wang","slug":"jianping-wang","given":"Jianping","family":"Wang","affiliation":"College of Computer Science and Software Engineering , Shenzhen University"},{"name":"Hang Guo","slug":"hang-guo","given":"Hang","family":"Guo","affiliation":"Tsinghua Shenzhen International Graduate School , Tsinghua University"},{"name":"Jinmin Li","slug":"jinmin-li","given":"Jinmin","family":"Li","affiliation":"Tsinghua Shenzhen International Graduate School , Tsinghua University"},{"name":"Jinbao Wang","slug":"jinbao-wang","given":"Jinbao","family":"Wang","affiliation":"College of Computer Science and Software Engineering , Shenzhen University; National Engineering Laboratory for Big Data System Computing Technology , Shenzhen University"},{"name":"Zexuan Zhu","slug":"zexuan-zhu","given":"Zexuan","family":"Zhu","affiliation":"College of Computer Science and Software Engineering , Shenzhen University; National Engineering Laboratory for Big Data System Computing Technology , Shenzhen University"}],"abstract":"Transformer-based models have been widely and successfully used in various low-vision visual  tasks,  and have achieved remarkable performance in single image super-resolution (SR). Despite the significant progress in SR, Transformer-based SR methods (e.g., SwinIR) still suffer from the problems of heavy computation cost and low-frequency preference, while ignoring the reconstruction of rich high-frequency information, hence hindering the representational power of Transformers. To address these issues, in this paper, we propose  a novel Frequency-aware Transformer (FreqFormer) for lightweight image SR. Specifically, a Frequency Division Module (FDM) is first introduced to separately handle  high- and low-frequency information in a divide-and-conquer manner. Moreover, we present  Frequency-aware Transformer Block (FTB) to extracting both spatial frequency attention  and channel transposed attention to recover high-frequency details. Extensive experimental results on public datasets demonstrate the superiority of our FreqFormer over state-of-the-art SR methods in terms of both quantitative metrics and visual quality. Code and models are available at https://github.com/JPWang-CS/FreqFormer.","doi":"10.24963/ijcai.2024/81","url":"https://ijcai.org/proceedings/2024/81","pdf":"https://ijcai.org/proceedings/2024/0081.pdf","markdown":"https://ijcai.org/proceedings/2024/81.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/81","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":81,"pages":{"first":731,"last":739},"keywords":["Computer Vision: CV: Machine learning for vision","Computer Vision: CV: Applications","Computer Vision: CV: Image and video synthesis and generation","Computer Vision: CV: Interpretability and transparency"],"videos":[],"references":[{"position":1,"raw":"[Bevilacqua et al., 2012] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Alberi Morel. Low-complexity single image super-resolution based on nonnegative neigh- bor embedding. In BMVC, 2012.","title":"Low-complexity single image super-resolution based on nonnegative neighbor embedding","authors":"Marco Bevilacqua, Aline Roumy, Christine Guillemot, Alberi Morel","year":"2012","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2023] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggre- gation transformer for image super-resolution. In CVPR, 2023.","title":"Dual aggregation transformer for image super-resolution","authors":"Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, Fisher Yu","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2024] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, and Xiaokang Yang. Recursive general- ization transformer for image super-resolution. In ICLR, 2024.","title":"Recursive generalization transformer for image super-resolution","authors":"Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang","year":"2024","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi et al., 2023] Haram Choi, Jeongmin Lee, and Ji- hoon Yang. N-gram in swin transformers for efficient lightweight image super-resolution. In CVPR, 2023.","title":"N-gram in swin transformers for efficient lightweight image super-resolution","authors":"Choi","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cui et al., 2024] Y. Cui, W. Ren, X. Cao, and A. Knoll. Image restoration via frequency selection. IEEE TPAMI, 2024.","title":"Image restoration via frequency selection","authors":"Y Cui, W Ren, X Cao, A Knoll","year":"2024","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[D. et al., 2002] Martin D., Fowlkes C., Tal D., and Malik J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2002.","title":"A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics","authors":"D Martin, C Fowlkes, D Tal, Malik J","year":"2002","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dai et al., 2019] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention net- work for single image super-resolution. In CVPR, 2019.","title":"Second-order attention network for single image super-resolution","authors":"Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, Lei Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2023] Tao Dai, Mengxi Ya, Jinmin Li, Xinyi Zhang, Shu-Tao Xia, and Zexuan Zhu. Cfgn: A lightweight context feature guided network for image super-resolution. IEEE TETCI, 2023.","title":"Cfgn: A lightweight context feature guided network for image super-resolution","authors":"Tao Dai, Mengxi Ya, Jinmin Li, Xinyi Zhang, Shu-Tao Xia, Zexuan Zhu","year":"2023","venue":"IEEE TETCI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong et al., 2016a] Chao Dong, Chen Change Loy, Kaim- ing He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE TPAMI, 2016.","title":"Image super-resolution using deep convolutional networks","authors":"Dong","year":"2016","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dong et al., 2016b] Chao Dong, Chen Change Loy, and Xi- aoou Tang. Accelerating the super-resolution convolu- tional neural network. In ECCV, 2016.","title":"Accelerating the super-resolution convolutional neural network","authors":"Dong","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guo et al., 2024] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: A sim- ple baseline for image restoration with state-space model. In arXiv, 2024.","title":"Mambair: A simple baseline for image restoration with state-space model","authors":"Guo","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2015] Jia Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015.","title":"Single image super-resolution from transformed self-exemplars","authors":"Jia Bin Huang, Abhishek Singh, Narendra Ahuja","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hui et al., 2019] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACM, MM, 2019.","title":"Lightweight image super-resolution with information multi-distillation network","authors":"Zheng Hui, Xinbo Gao, Yunchu Yang, Xiumei Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2016a] Jiwon Kim, Jung Kwon Lee, and Ky- oung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.","title":"Accurate image super-resolution using very deep convolutional networks","authors":"Kim","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2016b] Jiwon Kim, Jung Kwon Lee, and Ky- oung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In CVPR, 2016.","title":"Deeply-recursive convolutional network for image super-resolution","authors":"Kim","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. In CVPR, 2021.","title":"Localvit: Bringing locality to vision transformers","authors":"Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, Luc Van Gool","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2023a] Ao Li, Le Zhang, Yun Liu, and Ce Zhu. Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution. In ICCV, 2023.","title":"Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution","authors":"Li","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2023b] Jinmin Li, Tao Dai, Mingyan Zhu, Bin Chen, Zhi Wang, and Shu-Tao Xia. Fsr: A general frequency-oriented framework to accelerate image super- resolution networks. In AAAI, 2023.","title":"Fsr: A general frequency-oriented framework to accelerate image superresolution networks","authors":"Li","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In CVPR, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lim et al., 2017] B. Lim, S. Son, H. Kim, S. Nah, and K. Lee. Enhanced deep residual networks for single im- age super-resolution. In CVPRW, 2017.","title":"Enhanced deep residual networks for single image super-resolution","authors":"B Lim, S Son, H Kim, S Nah, K Lee","year":"2017","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lu et al., 2022] Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, and Tieyong Zeng. Trans- former for single image super-resolution. In CVPR, 2022.","title":"Transformer for single image super-resolution","authors":"Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, Tieyong Zeng","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Luo et al., 2020] Xiaotong Luo, Yuan Xie, Yulun Zhang, Yanyun Qu, Cuihua Li, and Yun Fu. Latticenet: Towards lightweight image super-resolution with lattice block. In ECCV, 2020.","title":"Latticenet: Towards lightweight image super-resolution with lattice block","authors":"Xiaotong Luo, Yuan Xie, Yulun Zhang, Yanyun Qu, Cuihua Li, Yun Fu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Matsui et al., 2016] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. IEEE MTA, 2016.","title":"Sketch-based manga retrieval using manga109 dataset","authors":"Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, Kiyoharu Aizawa","year":"2016","venue":"IEEE MTA","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Park and Kim, 2022] Namuk Park and Songkuk Kim. How do vision transformers work? In ICLR, 2022.","title":"Namuk Park and Songkuk Kim. How do vision transformers work?","authors":"Kim Park","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Patro and Agneeswaran, 2023] Badri N. Patro and Vi- jay Srinivas Agneeswaran. Scattering vision transformer: Spectral mixing matters. In NeurIPS, 2023.","title":"Scattering vision transformer: Spectral mixing matters","authors":"N Badri, Vijay Patro, Srinivas Agneeswaran","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Radu Timofte and Zhang, 2017] Luc Van Gool Ming- Hsuan Yang Radu Timofte, Eirikur Agustsson and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPRW, 2017.","title":"Ntire 2017 challenge on single image super-resolution: Methods and results","authors":"Radu Timofte, Luc Zhang, Ming-Hsuan Van Gool, Eirikur Yang Radu Timofte, Lei Agustsson, Zhang","year":"2017","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ren et al., 2019] Wenqi Ren, Jiaolong Yang, Senyou Deng, David Wipf, Xiaochun Cao, and Tong Xin. Face video deblurring using a 3d facial prior. In ICCV, 2019.","title":"Face video deblurring using a 3d facial prior","authors":"Jiaolong Wenqi Ren, Senyou Yang, David Deng, Xiaochun Wipf, Tong Cao, Xin","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shi et al., 2016] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single im- age and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.","title":"Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network","authors":"Shi","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2023] Bin Sun, Yulun Zhang, Songyao Jiang, and Yun Fu. Hybrid pixel-unshuffled network for lightweight image super-resolution. In AAAI, 2023.","title":"Hybrid pixel-unshuffled network for lightweight image super-resolution","authors":"Bin Sun, Yulun Zhang, Songyao Jiang, Yun Fu","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tai et al., 2017] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In ICCV, 2017.","title":"Memnet: A persistent memory network for image restoration","authors":"Ying Tai, Jian Yang, Xiaoming Liu, Chunyan Xu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2020] Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. IEEE TPAMI, 2020.","title":"Deep learning for image super-resolution: A survey","authors":"Zhihao Wang, Jian Chen, Steven Ch Hoi","year":"2020","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In ICCV, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2022] Zhendong Wang, Xiaodong Cun, Jian- min Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022.","title":"Uformer: A general u-shaped transformer for image restoration","authors":"Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, Houqiang Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2020] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Va- jda. Visual transformers: Token-based image representa- tion and processing for computer vision. In CVPR, 2020.","title":"Visual transformers: Token-based image representation and processing for computer vision","authors":"Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, Peter Vajda","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zeyde et al., 2010] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In ICCS, 2010.","title":"On single image scale-up using sparserepresentations","authors":"Zeyde","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2018] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. In ECCV, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2022] Xindong Zhang, Hui Zeng, Shi Guo, and Lei Zhang. Efficient long-range attention network for image super-resolution. In ECCV, 2022.","title":"Efficient long-range attention network for image super-resolution","authors":"Xindong Zhang, Hui Zeng, Shi Guo, Lei Zhang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang et al., 2023] Aiping Zhang, Wenqi Ren, Yi Liu, and Xiaochun Cao. Lightweight image super-resolution with superpixel token interaction. In ICCV, 2023.","title":"Lightweight image super-resolution with superpixel token interaction","authors":"Aiping Zhang, Wenqi Ren, Yi Liu, Xiaochun Cao","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhou et al., 2023] Yupeng Zhou, Zhen Li, Chun-Le Guo, Song Bai, Ming-Ming Cheng, and Qibin Hou. Srformer: Permuted self-attention for single image super-resolution. In ICCV, 2023.","title":"Srformer: Permuted self-attention for single image super-resolution","authors":"Yupeng Zhou, Zhen Li, Chun-Le Guo, Song Bai, Ming-Ming Cheng, Qibin Hou","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}