{"type":"ScholarlyArticle","title":"IDPT: Interconnected Dual Pyramid Transformer for Face Super-Resolution","authors":[{"name":"Jingang Shi","slug":"jingang-shi","given":"Jingang","family":"Shi","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Yusi Wang","slug":"yusi-wang","given":"Yusi","family":"Wang","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Songlin Dong","slug":"songlin-dong","given":"Songlin","family":"Dong","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Xiaopeng Hong","slug":"xiaopeng-hong","given":"Xiaopeng","family":"Hong","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Zitong Yu","slug":"zitong-yu","given":"Zitong","family":"Yu","affiliation":"School of Electrical and Electronic Engineering , Nanyang Technological University"},{"name":"Fei Wang","slug":"fei-wang","given":"Fei","family":"Wang","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Changxin Wang","slug":"changxin-wang","given":"Changxin","family":"Wang","affiliation":"School of Software Engineering , Xi'an Jiaotong University"},{"name":"Yihong Gong","slug":"yihong-gong","given":"Yihong","family":"Gong","affiliation":"School of Software Engineering , Xi'an Jiaotong University"}],"abstract":"Face Super-resolution (FSR) task works for generating high-resolution (HR) face images from the corresponding low-resolution (LR) inputs, which has received a lot of attentions because of the wide application prospects. However, due to the diversity of facial texture and the difficulty of reconstructing detailed content from degraded images, FSR technology is still far away from being solved. In this paper, we propose a novel and effective face super-resolution framework based on Transformer, namely Interconnected Dual Pyramid Transformer (IDPT). Instead of straightly stacking cascaded feature reconstruction blocks, the proposed IDPT designs the pyramid encoder/decoder Transformer architecture to extract coarse and detailed facial textures respectively, while the relationship between the dual pyramid Transformers is further explored by a bottom pyramid feature extractor. The pyramid encoder/decoder structure is devised to adapt various characteristics of textures in different spatial spaces hierarchically. A novel fusing modulation module is inserted in each spatial layer to guide the refinement of detailed texture by the corresponding coarse texture, while fusing the shallow-layer coarse feature and corresponding deep-layer detailed feature simultaneously. Extensive experiments and visualizations on various datasets demonstrate the superiority of the proposed method for face super-resolution tasks.","doi":"10.24963/ijcai.2022/182","url":"https://ijcai.org/proceedings/2022/182","pdf":"https://ijcai.org/proceedings/2022/0182.pdf","markdown":"https://ijcai.org/proceedings/2022/182.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/182","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":182,"pages":{"first":1306,"last":1312},"keywords":["Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Computer Vision: Computational photography"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986512","length_sec":67,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984655","length_sec":321,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cao et al., 2021] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool. Video super-resolution transformer. arXiv preprint arXiv:2106.06847, 2021.","title":"Video super-resolution transformer","authors":"Cao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Yu Chen, Ying Tai, Xiaoming Liu, Chun- hua Shen, and Jian Yang. FSRNet: End-to-end learning face super-resolution with facial priors. In CVPR, pages 2492-2501, 2018.","title":"FSRNet: End-to-end learning face super-resolution with facial priors","authors":"Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, Jian Yang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, and Kwan-Yee Wong. Learning spa- tial attention for face super-resolution. IEEE Transactions on Image Processing, 30:1219-1231, 2020.","title":"Learning spatial attention for face super-resolution","authors":"Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, Kwan-Yee Wong","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, and et.al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Et","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jiang et al., 2021] Junjun Jiang, Chenyang Wang, Xian- ming Liu, and Jiayi Ma. Deep learning-based face super- resolution: A survey. ACM Computing Surveys, 55(1):1- 36, 2021.","title":"Deep learning-based face superresolution: A survey","authors":"Junjun Jiang, Chenyang Wang, Xianming Liu, Jiayi Ma","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Khan et al., 2021] Salman Khan, Muzammal Naseer, Mu- nawar Hayat, and et al. Transformers in vision: A survey. arXiv preprint arXiv:2101.01169, 2021.","title":"Transformers in vision: A survey","authors":"Salman Khan, Muzammal Naseer, Munawar Hayat","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim et al., 2019] Deokyun Kim, Minseon Kim, Gihyun Kwon, and et al. Progressive face super-resolution via at- tention to face landmark. In BMVC, 2019.","title":"Progressive face super-resolution via attention to face landmark","authors":"Kim","year":"2019","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Le et al., 2012] Vuong Le, Jonathan Brandt, Zhe Lin, and et al. Interactive facial feature localization. In ECCV, pages 679-692. Springer, 2012.","title":"Interactive facial feature localization","authors":"Jonathan Vuong Le, Zhe Brandt, Lin","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszár, and et al. Photo-realistic single image super- resolution using a generative adversarial network. In CVPR, pages 4681-4690, 2017.","title":"Photo-realistic single image superresolution using a generative adversarial network","authors":"Ledig","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, and et al. SwinIR: Image restoration using swin trans- former. In CVPR, pages 1833-1844, 2021.","title":"SwinIR: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2017] Ling Liu, Fan Wang, Ke Zhou, Nai Ding, and Huan Luo. Perceptual integration rapidly activates dorsal visual pathway to guide local processing in early visual areas. PLoS biology, 15(11):e2003646, 2017.","title":"Perceptual integration rapidly activates dorsal visual pathway to guide local processing in early visual areas","authors":"Ling Liu, Fan Wang, Ke Zhou, Nai Ding, Huan Luo","year":"2017","venue":"PLoS biology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, and et al. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2021] Tao Lu, Yuanzhi Wang, Yanduo Zhang, and et al. Face hallucination via split-attention in split- attention network. In ACM MM, pages 5501-5509, 2021.","title":"Face hallucination via split-attention in splitattention network","authors":"Tao Lu, Yuanzhi Wang, Yanduo Zhang","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2020] Cheng Ma, Zhenyu Jiang, Yongming Rao, and et al. Deep face super-resolution with iterative collab- oration between attentive recovery and landmark estima- tion. In CVPR, pages 5569-5578, 2020.","title":"Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation","authors":"Cheng Ma, Zhenyu Jiang, Yongming Rao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shi and Zhao, 2019] Jingang Shi and Guoying Zhao. Face hallucination via coarse-to-fine recursive kernel regression structure. IEEE Transactions on Multimedia, 21(9):2223- 2236, 2019.","title":"Face hallucination via coarse-to-fine recursive kernel regression structure","authors":"Zhao ; ] Jingang Shi, Guoying Shi, Zhao","year":"2019","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2018] Jingang Shi, Xin Liu, Yuan Zong, Chun Qi, and Guoying Zhao. Hallucinating face image by regularization models in high-resolution feature space. IEEE Transactions on Image Processing, 27(6):2980- 2995, 2018.","title":"Hallucinating face image by regularization models in high-resolution feature space","authors":"Jingang Shi, Xin Liu, Yuan Zong, Chun Qi, Guoying Zhao","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu et al., 2018] Xiang Wu, Ran He, Zhenan Sun, and Tie- niu Tan. A light cnn for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security, 13(11):2884-2896, 2018.","title":"A light cnn for deep face representation with noisy labels","authors":"Xiang Wu, Ran He, Zhenan Sun, Tieniu Tan","year":"2018","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2021] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. GAN prior embedded network for blind face restoration in the wild. In CVPR, pages 672-681, 2021.","title":"GAN prior embedded network for blind face restoration in the wild","authors":"Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yu and Porikli, 2016] Xin Yu and Fatih Porikli. Ultra- resolving face images by discriminative generative net- works. In ECCV, pages 318-333. Springer, 2016.","title":"Ultraresolving face images by discriminative generative networks","authors":"Porikli ; Yu, Xin Yu, Fatih Porikli","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yu et al., 2018] Xin Yu, Basura Fernando, Bernard Ghanem, Fatih Porikli, and Richard Hartley. Face super-resolution guided by facial component heatmaps. In ECCV, pages 217-233, 2018.","title":"Face super-resolution guided by facial component heatmaps","authors":"Xin Yu, Basura Fernando, Bernard Ghanem, Fatih Porikli, Richard Hartley","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2022] Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Philip Torr, and Guoying Zhao. Phys- former: Facial video-based physiological measurement with temporal difference transformer. In CVPR, 2022.","title":"Physformer: Facial video-based physiological measurement with temporal difference transformer","authors":"Yu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yuan et al., 2021] Li Yuan, Yunpeng Chen, Tao Wang, Wei- hao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, pages 558-567, 2021.","title":"Tokens-to-token vit: Training vision transformers from scratch on imagenet","authors":"Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis Eh Tay, Jiashi Feng, Shuicheng Yan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2016] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and align- ment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499-1503, 2016.","title":"Joint face detection and alignment using multitask cascaded convolutional networks","authors":"Zhang","year":"2016","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2018] Kaipeng Zhang, Zhanpeng Zhang, Chia-Wen Cheng, Winston Hsu, Yu Qiao, Wei Liu, and Tong Zhang. Super-identity convolutional neural network for face hallucination. In ECCV, pages 183-198, 2018.","title":"Super-identity convolutional neural network for face hallucination","authors":"Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}