{"type":"ScholarlyArticle","title":"Beyond Alignment: Blind Video Face Restoration via Parsing-Guided Temporal-Coherent Transformer","authors":[{"name":"Kepeng Xu","slug":"kepeng-xu","given":"Kepeng","family":"Xu","affiliation":"Xidian University"},{"name":"Li Xu","slug":"li-xu","given":"Li","family":"Xu","affiliation":"Xidian University"},{"name":"Gang He","slug":"gang-he","given":"Gang","family":"He","affiliation":"Xidian University"},{"name":"Wenxin Yu","slug":"wenxin-yu","given":"Wenxin","family":"Yu","affiliation":"Southwest University of Science and Technology"},{"name":"Yunsong Li","slug":"yunsong-li","given":"Yunsong","family":"Li","affiliation":"Xidian University"}],"abstract":"Multiple complex degradations are coupled in low-quality video faces in the real world. Therefore, blind video face restoration is a highly challenging ill-posed problem, requiring not only hallucinating high-fidelity details but also enhancing temporal coherence across diverse pose variations. Restoring each frame independently in a naive manner inevitably introduces temporal incoherence and artifacts from pose changes and keypoint localization errors. To address this, we propose the first blind video face restoration approach with a novel parsing-guided temporal-coherent transformer (PGTFormer) without pre-alignment. PGTFormer leverages semantic parsing guidance to select optimal face priors for generating temporally coherent artifact-free results. Specifically, we pre-train a temporal-spatial vector quantized auto-encoder on high-quality video face datasets to extract expressive context-rich priors. Then, the temporal parse-guided codebook predictor (TPCP) restores faces in different poses based on face parsing context cues without performing face pre-alignment. This strategy reduces artifacts and mitigates jitter caused by cumulative errors from face pre-alignment. Finally, the temporal fidelity regulator (TFR) enhances fidelity through temporal feature interaction and improves video temporal consistency. Extensive experiments on face videos show that our method outperforms previous face restoration baselines. The code will be released on https://github.com/kepengxu/PGTFormer.","doi":"10.24963/ijcai.2024/165","url":"https://ijcai.org/proceedings/2024/165","pdf":"https://ijcai.org/proceedings/2024/0165.pdf","markdown":"https://ijcai.org/proceedings/2024/165.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/165","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":165,"pages":{"first":1489,"last":1497},"keywords":["Computer Vision: CV: Image and video synthesis and generation","Computer Vision: CV: Computational photography","Computer Vision: CV: Applications","Computer Vision: CV: Biometrics, face, gesture and pose recognition"],"videos":[],"references":[{"position":1,"raw":"[Chan et al., 2021a] Kelvin C.K. Chan, Xintao Wang, Xi- angyu Xu, Jinwei Gu, and Chen Change Loy. Glean: Gen- erative latent bank for large-factor image super-resolution. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.","title":"Glean: Generative latent bank for large-factor image super-resolution","authors":"Chan","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chan et al., 2021b] Kelvin C.K. Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Basicvsr: The search for essential components in video super-resolution and beyond. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.","title":"Basicvsr: The search for essential components in video super-resolution and beyond","authors":"Chan","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chan et al., 2022] Kelvin C.K. Chan, Shangchen Zhou, Xi- angyu Xu, and Chen Change Loy. Basicvsr++: Improv- ing video super-resolution with enhanced propagation and alignment. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022.","title":"Basicvsr++: Improving video super-resolution with enhanced propagation and alignment","authors":"C K Kelvin, Shangchen Chan, Xiangyu Zhou, Chen Change Xu, Loy","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Yu Chen, Ying Tai, Xiaoming Liu, Chun- hua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018.","title":"Fsrnet: End-to-end learning face super-resolution with facial priors","authors":"Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, Jian Yang","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K. Wong. Progressive semantic-aware style transformation for blind face restoration. In 2021 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), Jun 2021.","title":"Progressive semantic-aware style transformation for blind face restoration","authors":"Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, K Kwan-Yee, Wong","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2024] Wei Chen, Yuxuan Liu, Zhao Zhang, Fuzhen Zhuang, and Jiang Zhong. Modeling adap- tive inter-task feature interactions via sentiment-aware contrastive learning for joint aspect-sentiment prediction. 38:17781-17789, 2024.","title":"Modeling adaptive inter-task feature interactions via sentiment-aware contrastive learning for joint aspect-sentiment prediction","authors":"Wei Chen, Yuxuan Liu, Zhao Zhang, Fuzhen Zhuang, Jiang Zhong","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dai et al., 2022] Peng Dai, Xin Yu, Lan Ma, Baoheng Zhang, Jia Li, Wenbo Li, Jiajun Shen, and Xiaojuan Qi. Video demoireing with relation-based temporal consis- tency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17622- 17631, 2022.","title":"Video demoireing with relation-based temporal consistency","authors":"Peng Dai, Xin Yu, Lan Ma, Baoheng Zhang, Jia Li, Wenbo Li, Jiajun Shen, Xiaojuan Qi","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2021] Jiankang Deng, Jia Guo, Jing Yang, Ni- annan Xue, Irene Cotsia, and Stefanos P Zafeiriou. Arc- face: Additive angular margin loss for deep face recogni- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1-1, Jan 2021.","title":"Arcface: Additive angular margin loss for deep face recognition","authors":"Deng","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2020] Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020.","title":"Image processing using multi-code gan prior","authors":"Jinjin Gu, Yujun Shen, Bolei Zhou","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2022] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. Vqfr: Blind face restoration with vector-quantized dictio- nary and parallel decoder. In ECCV, 2022.","title":"Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder","authors":"Gu","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guan et al., 2021] Zhenyu Guan, Qunliang Xing, Mai Xu, Ren Yang, Tie Liu, and Zulin Wang. Mfqe 2.0: A new approach for multi-frame quality enhancement on com- pressed video. IEEE Transactions on Pattern Analysis and Machine Intelligence, page 949-963, Mar 2021.","title":"Mfqe 2.0: A new approach for multi-frame quality enhancement on compressed video","authors":"Qunliang Zhenyu Guan, Mai Xing, Ren Xu, Tie Yang, Zulin Liu, Wang","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2022] Gang He, Kepeng Xu, Li Xu, Chang Wu, Ming Sun, Xing Wen, and Yu-Wing Tai. Sdrtv-to-hdrtv via hierarchical dynamic context feature mapping. page 2890-2898, 2022.","title":"Sdrtv-to-hdrtv via hierarchical dynamic context feature mapping","authors":"Gang He, Kepeng Xu, Li Xu, Chang Wu, Ming Sun, Xing Wen, Yu-Wing Tai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2020] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2020.","title":"Maskgan: Towards diverse and interactive facial image manipulation","authors":"Ziwei Cheng-Han Lee, Lingyun Liu, Ping Wu, Luo","year":"2020","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Siyuan Li, Lu Lu, Zhiqiang Zhang, Xin Cheng, Kepeng Xu, Wenxin Yu, Gang He, Jinjia Zhou, and Zhuo Yang. Interactive separation network for image inpainting. pages 1008-1012, 2020.","title":"Interactive separation network for image inpainting","authors":"Siyuan Li, Lu Lu, Zhiqiang Zhang, Xin Cheng, Kepeng Xu, Wenxin Yu, Gang He, Jinjia Zhou, Zhuo Yang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2022] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202- 3211, 2022.","title":"Video swin transformer","authors":"Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, Han Hu","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Menon et al., 2020] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020.","title":"Pulse: Selfsupervised photo upsampling via latent space exploration of generative models","authors":"Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, Cynthia Rudin","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Merritt and Vanam, 2006] Loren Merritt and Rahul Vanam. x264: A high performance h. 264/avc encoder. online http://neuron2. net/library/avc/overview x264 v8 5. pdf, 2006.","title":"A high performance h. 264/avc encoder","authors":"Loren Merritt, Rahul Vanam","year":"2006","venue":null,"doi":null,"url":"http://neuron2.net/library/avc/overviewx264v85.pdf","internal_url":null},{"position":18,"raw":"[Sun et al., 2020] Yuan Sun, Zhenwen Ren, Chao Yang, Quansen Sun, Liwan Chen, and Yanglong Ou. Face im- age set classification with self-weighted latent sparse dis- criminative learning. Neural Computing and Applications, pages 1-13, 2020.","title":"Face image set classification with self-weighted latent sparse discriminative learning","authors":"Yuan Sun, Zhenwen Ren, Chao Yang, Quansen Sun, Liwan Chen, Yanglong Ou","year":"2020","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2019a] Xintao Wang, Kelvin C.K. Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional net- works. In 2019 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition Workshops (CVPRW), Jun 2019.","title":"Edvr: Video restoration with enhanced deformable convolutional networks","authors":"Wang","year":"2019","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019b] Xinyao Wang, Liefeng Bo, and Li Fuxin. Adaptive wing loss for robust face alignment via heatmap regression. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019.","title":"Adaptive wing loss for robust face alignment via heatmap regression","authors":"Wang","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restora- tion with generative facial prior. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.","title":"Towards real-world blind face restoration with generative facial prior","authors":"Wang","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2024a] Cong Wang, Jinshan Pan, Wanyu Lin, Jiangxin Dong, Wei Wang, and Xiao-Ming Wu. Self- promer: Self-prompt dehazing transformers with depth- consistency. 38:5327-5335, 2024.","title":"Selfpromer: Self-prompt dehazing transformers with depthconsistency","authors":"Wang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2024b] Cong Wang, Jinshan Pan, Wei Wang, Jiangxin Dong, Mengzhu Wang, Yakun Ju, and Junyang Chen. Promptrestorer: A prompting image restoration method with degradation perception. Advances in Neural Information Processing Systems, 36, 2024.","title":"Promptrestorer: A prompting image restoration method with degradation perception","authors":"Wang","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2024c] Cong Wang, Jinshan Pan, Wei Wang, Gang Fu, Siyuan Liang, Mengzhu Wang, Xiao-Ming Wu, and Jun Liu. Correlation matching transformation trans- formers for uhd image restoration. 38:5336-5344, 2024.","title":"Correlation matching transformation transformers for uhd image restoration","authors":"Wang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2022] Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022.","title":"Vfhq: A high-quality dataset and benchmark for video face super-resolution","authors":"Xie","year":"2022","venue":"The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2022a] Kepeng Xu, Li Xu, Gang He, Chang Wu, Zijia Ma, Ming Sun, and Yu-Wing Tai. Sdrtv-to-hdrtv con- version via spatial-temporal feature fusion. arXiv preprint arXiv:2211.02297, 2022.","title":"Sdrtv-to-hdrtv conversion via spatial-temporal feature fusion","authors":"Xu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2022b] Li Xu, Gang He, Jinjia Zhou, Jie Lei, Weiying Xie, Yunsong Li, and Yu-Wing Tai. Transcoded video restoration by temporal spatial auxiliary network. 36(3):2875-2883, 2022.","title":"Transcoded video restoration by temporal spatial auxiliary network","authors":"Xu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2023] Kepeng Xu, Gang He, Li Xu, Xingchao Yang, Ming Sun, Yuzhi Wang, Zijia Ma, Haoqiang Fan, and Xing Wen. Towards robust sdrtv-to-hdrtv via dual inverse degradation network. arXiv preprint arXiv:2307.03394, 2023.","title":"Towards robust sdrtv-to-hdrtv via dual inverse degradation network","authors":"Kepeng Xu, Gang He, Li Xu, Xingchao Yang, Ming Sun, Yuzhi Wang, Zijia Ma, Haoqiang Fan, Xing Wen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2018] Ren Yang, Mai Xu, Zulin Wang, and Tianyi Li. Multi-frame quality enhancement for com- pressed video. In 2018 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, Jun 2018.","title":"Multi-frame quality enhancement for compressed video","authors":"Ren Yang, Mai Xu, Zulin Wang, Tianyi Li","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2021] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.","title":"Gan prior embedded network for blind face restoration in the wild","authors":"Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2021] Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang. Bisenet v2: Bilateral network with guided aggregation for real- time semantic segmentation. International Journal of Computer Vision, 129:3051-3068, 2021.","title":"Bisenet v2: Bilateral network with guided aggregation for realtime semantic segmentation","authors":"Yu","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2024] Jiaqing Zhang, Mingxiang Cao, Xue Yang, Weiying Xie, Jie Lei, Daixun Li, Geng Yang, Wenbo Huang, and Yunsong Li. Efficientmfd: Towards more ef- ficient multimodal synchronous fusion detection. arXiv preprint arXiv:2403.09323, 2024.","title":"Efficientmfd: Towards more efficient multimodal synchronous fusion detection","authors":"Jiaqing Zhang, Mingxiang Cao, Xue Yang, Weiying Xie, Jie Lei, Daixun Li, Geng Yang, Wenbo Huang, Yunsong Li","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2022] Shangchen Zhou, KelvinC.K. Chan, Chongyi Li, and ChenChange Loy. Towards robust blind face restoration with codebook lookup transformer. NeurIPS, Jun 2022.","title":"Towards robust blind face restoration with codebook lookup transformer","authors":"Zhou","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}