{"type":"ScholarlyArticle","title":"Learning Attention from Attention: Efficient Self-Refinement Transformer for Face Super-Resolution","authors":[{"name":"Guanxin Li","slug":"guanxin-li","given":"Guanxin","family":"Li","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Jingang Shi","slug":"jingang-shi","given":"Jingang","family":"Shi","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Yuan Zong","slug":"yuan-zong","given":"Yuan","family":"Zong","affiliation":"Key Laboratory of Child Development and Learning Science , Southeast University"},{"name":"Fei Wang","slug":"fei-wang","given":"Fei","family":"Wang","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Tian Wang","slug":"tian-wang","given":"Tian","family":"Wang","affiliation":"Institute of Artificial Intelligence , Beihang University"},{"name":"Yihong Gong","slug":"yihong-gong","given":"Yihong","family":"Gong","affiliation":"School of Software Engineering , Xi'an Jiaotong University"}],"abstract":"Recently, Transformer-based architecture has been introduced into face super-resolution task due to its advantage in capturing long-range dependencies. However, these approaches tend to integrate global information in a large searching region, which neglect to focus on the most relevant information and induce blurry effect by the irrelevant textures. Some improved methods simply constrain self-attention in a local window to suppress the useless information. But it also limits the capability of recovering high-frequency details when flat areas dominate the local searching window. To improve the above issues, we propose a novel self-refinement mechanism which could adaptively achieve texture-aware reconstruction in a coarse-to-fine procedure. Generally, the primary self-attention is first conducted to reconstruct the coarse-grained textures and detect the fine-grained regions required further compensation. Then, region selection attention is performed to refine the textures on these key regions. Since self-attention considers the channel information on tokens equally, we employ a dual-branch feature integration module to privilege the important channels in feature extraction. Furthermore, we design the wavelet fusion module which integrate shallow-layer structure and deep-layer detailed feature to recover realistic face images in frequency domain. Extensive experiments demonstrate the effectiveness on a variety of datasets.","doi":"10.24963/ijcai.2023/115","url":"https://ijcai.org/proceedings/2023/115","pdf":"https://ijcai.org/proceedings/2023/0115.pdf","markdown":"https://ijcai.org/proceedings/2023/115.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/115","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":115,"pages":{"first":1035,"last":1043},"keywords":["Computer Vision: CV: Computational photography","Computer Vision: CV: Biometrics, face, gesture and pose recognition"],"videos":[],"references":[{"position":1,"raw":"[Bulat and Tzimiropoulos, 2018] Adrian Bulat and Georgios Tzimiropoulos. Super-fan: Integrated facial landmark lo- calization and super-resolution of real-world low resolu- tion faces in arbitrary poses with gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 109-117, 2018.","title":"Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans","authors":"Adrian Bulat, Georgios Tzimiropoulos","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2017] Qingxing Cao, Liang Lin, Yukai Shi, Xi- aodan Liang, and Guanbin Li. Attention-aware face hallu- cination via deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 690-698, 2017.","title":"Attention-aware face hallucination via deep reinforcement learning","authors":"Qingxing Cao, Liang Lin, Yukai Shi, Xiaodan Liang, Guanbin Li","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2021] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool. Video super-resolution transformer. arXiv preprint arXiv:2106.06847, 2021.","title":"Video super-resolution transformer","authors":"Jiezhang Cao, Yawei Li, Kai Zhang, Luc Van Gool","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Yu Chen, Ying Tai, Xiaoming Liu, Chun- hua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 2492-2501, 2018.","title":"Fsrnet: End-to-end learning face super-resolution with facial priors","authors":"Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, Jian Yang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, and Kwan-Yee K Wong. Learning spa- tial attention for face super-resolution. IEEE Transactions on Image Processing, 30:1219-1231, 2020.","title":"Learning spatial attention for face super-resolution","authors":"Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, Kwan-Yee K Wong","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2021] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural In- formation Processing Systems, 34, 2021.","title":"Twins: Revisiting the design of spatial attention in vision transformers","authors":"Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, Chunhua Shen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In Proceedings of the IEEE International Conference on Computer Vision, pages 1689-1697, 2017.","title":"Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution","authors":"Huang","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Le et al., 2012] Vuong Le, Jonathan Brandt, Zhe Lin, Lubomir Bourdev, and Thomas S Huang. Interactive facial feature localization. In European conference on computer vision, pages 679-692. Springer, 2012.","title":"Interactive facial feature localization","authors":"Jonathan Vuong Le, Zhe Brandt, Lubomir Lin, Thomas S Bourdev, Huang","year":"2012","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Zhuangzi Li, Ge Li, Thomas Li, Shan Liu, and Wei Gao. Information-growth attention network for image super-resolution. In Proceedings of the 29th ACM International Conference on Multimedia, pages 544-552, 2021.","title":"Information-growth attention network for image super-resolution","authors":"Zhuangzi Li, Ge Li, Thomas Li, Shan Liu, Wei Gao","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833-1844, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2020] Cheng Ma, Zhenyu Jiang, Yongming Rao, Jiwen Lu, and Jie Zhou. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5569-5578, 2020.","title":"Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation","authors":"Cheng Ma, Zhenyu Jiang, Yongming Rao, Jiwen Lu, Jie Zhou","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Maeda, 2022] Shunta Maeda. Image super-resolution with deep dictionary. In European Conference on Computer Vision, pages 464-480. Springer, 2022.","title":"Shunta Maeda. Image super-resolution with deep dictionary","authors":"Maeda","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shi and Zhao, 2019] Jingang Shi and Guoying Zhao. Face hallucination via coarse-to-fine recursive kernel regression structure. IEEE Transactions on Multimedia, 21(9):2223- 2236, 2019.","title":"Face hallucination via coarse-to-fine recursive kernel regression structure","authors":"Zhao ; ] Jingang Shi, Guoying Shi, Zhao","year":"2019","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2022] Jingang Shi, Yusi Wang, Songlin Dong, Xi- aopeng Hong, Zitong Yu, Fei Wang, Changxin Wang, and Yihong Gong. IDPT: interconnected dual pyramid trans- former for face super-resolution. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), pages 1306-1312, 2022.","title":"IDPT: interconnected dual pyramid transformer for face super-resolution","authors":"Jingang Shi, Yusi Wang, Songlin Dong, Xiaopeng Hong, Zitong Yu, Fei Wang, Changxin Wang, Yihong Gong","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2022/182"},{"position":18,"raw":"[Tian et al., 2022] Chunwei Tian, Yanning Zhang, Wang- meng Zuo, Chia-Wen Lin, David Zhang, and Yixuan Yuan. A heterogeneous group cnn for image super-resolution. IEEE Transactions on Neural Networks and Learning Sys- tems, 2022.","title":"A heterogeneous group cnn for image super-resolution","authors":"Chunwei Tian, Yanning Zhang, Wangmeng Zuo, Chia-Wen Lin, David Zhang, Yixuan Yuan","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tuzel et al., 2016] Oncel Tuzel, Yuichi Taguchi, and John R Hershey. Global-local face upsampling network. arXiv preprint arXiv:1603.07235, 2016.","title":"Global-local face upsampling network","authors":"Tuzel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021] Yu-Huan Wu, Yun Liu, Xin Zhan, and Ming-Ming Cheng. P2t: Pyramid pooling transformer for scene understanding. arXiv preprint arXiv:2106.12011, 2021.","title":"P2t: Pyramid pooling transformer for scene understanding","authors":"Yu-Huan Wu, Yun Liu, Xin Zhan, Ming-Ming Cheng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yuan et al., 2021] Li Yuan, Yunpeng Chen, Tao Wang, Wei- hao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 558-567, 2021.","title":"Tokens-to-token vit: Training vision transformers from scratch on imagenet","authors":"Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis Eh Tay, Jiashi Feng, Shuicheng Yan","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2016] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and align- ment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499-1503, 2016.","title":"Joint face detection and alignment using multitask cascaded convolutional networks","authors":"Zhang","year":"2016","venue":"IEEE signal processing letters","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018a] Kaipeng Zhang, Zhanpeng Zhang, Chia-Wen Cheng, Winston H Hsu, Yu Qiao, Wei Liu, and Tong Zhang. Super-identity convolutional neural network for face hallucination. In Proceedings of the European conference on computer vision (ECCV), pages 183-198, 2018.","title":"Super-identity convolutional neural network for face hallucination","authors":"Zhang","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018b] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Zhang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018c] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. In Proceedings of the European conference on computer vision (ECCV), pages 286-301, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}