{"type":"ScholarlyArticle","title":"RePFormer: Refinement Pyramid Transformer for Robust Facial Landmark Detection","authors":[{"name":"Jinpeng Li","slug":"jinpeng-li","given":"Jinpeng","family":"Li","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , Hong Kong , China"},{"name":"Haibo Jin","slug":"haibo-jin","given":"Haibo","family":"Jin","affiliation":"Department of Computer Science and Engineering , The Hong Kong University of Science and Technology , Hong Kong , China"},{"name":"Shengcai Liao","slug":"shengcai-liao","given":"Shengcai","family":"Liao","affiliation":"Inception Institute of Artificial Intelligence (IIAI) , UAE"},{"name":"Ling Shao","slug":"ling-shao","given":"Ling","family":"Shao","affiliation":"Terminus Group , China"},{"name":"Pheng-Ann Heng","slug":"pheng-ann-heng","given":"Pheng-Ann","family":"Heng","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , Hong Kong , China; Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology , Shenzhen Institutes of Advanced Technology , Chinese Academy of Sciences , Shenzhen , China"}],"abstract":"This paper presents a Refinement Pyramid Transformer (RePFormer) for robust facial landmark detection. Most facial landmark detectors focus on learning  representative image features. However, these CNN-based feature representations are not robust enough to handle complex real-world scenarios due to ignoring the internal structure of landmarks, as well as the relations between landmarks and context. In this work, we formulate the facial landmark detection task as refining landmark queries along pyramid memories. Specifically, a pyramid transformer head (PTH) is introduced to build both homologous relations among landmarks and heterologous relations between landmarks and cross-scale contexts. Besides, a dynamic landmark refinement (DLR) module is designed to decompose the landmark regression into an end-to-end refinement procedure, where the dynamically aggregated queries are transformed to residual coordinates predictions. Extensive experimental results on four facial landmark detection benchmarks and their various subsets demonstrate the superior performance and high robustness of our framework.","doi":"10.24963/ijcai.2022/152","url":"https://ijcai.org/proceedings/2022/152","pdf":"https://ijcai.org/proceedings/2022/0152.pdf","markdown":"https://ijcai.org/proceedings/2022/152.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/152","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":152,"pages":{"first":1088,"last":1094},"keywords":["Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987096","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985239","length_sec":677,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Burgos-Artizzu et al., 2013] Xavier P. Burgos-Artizzu, Pietro Perona, and Piotr Dollár. Robust face landmark estimation under occlusion. In ICCV, 2013.","title":"Robust face landmark estimation under occlusion","authors":"P Xavier, Pietro Burgos-Artizzu, Piotr Perona, Dollár","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dapogny et al., 2019] Arnaud Dapogny, Kevin Bailly, and Matthieu Cord. Decafa: Deep convolutional cascade for face alignment in the wild. In ICCV, 2019.","title":"Decafa: Deep convolutional cascade for face alignment in the wild","authors":"Arnaud Dapogny, Kevin Bailly, Matthieu Cord","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong and Yang, 2019] Xuanyi Dong and Yi Yang. Teacher supervises students how to learn from partially labeled im- ages for facial landmark detection. In ICCV, 2019.","title":"Teacher supervises students how to learn from partially labeled images for facial landmark detection","authors":"Dong, Yang ;, Xuanyi Dong, Yi Yang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng et al., 2018] Zhen-Hua Feng, Josef Kittler, Muham- mad Awais, Patrik Huber, and Xiao-Jun Wu. Wing loss for robust facial landmark localisation with convolutional neural networks. In CVPR, 2018.","title":"Wing loss for robust facial landmark localisation with convolutional neural networks","authors":"Zhen-Hua Feng, Josef Kittler, Muhammad Awais, Patrik Huber, Xiao-Jun Wu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2021] Haibo Jin, Shengcai Liao, and Ling Shao. Pixel-in-pixel net: Towards efficient facial landmark de- tection in the wild. Int. J. Comput. Vis., 2021.","title":"Pixel-in-pixel net: Towards efficient facial landmark detection in the wild","authors":"Haibo Jin, Shengcai Liao, Ling Shao","year":"2021","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ko, 2018] ByoungChul Ko. A brief review of facial emotion recognition based on visual information. Sensors, 2018.","title":"ByoungChul Ko. A brief review of facial emotion recognition based on visual information","authors":"Ko","year":"2018","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koestinger et al., 2011] Martin Koestinger, Paul Wohlhart, Peter M. Roth, and Horst Bischof. Annotated facial land- marks in the wild: A large-scale, real-world database for facial landmark localization. In Workshop on BeFIT, 2011.","title":"Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization","authors":"Martin Koestinger, Paul Wohlhart, Peter M Roth, Horst Bischof","year":"2011","venue":"Workshop on BeFIT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kumar et al., 2020] Abhinav Kumar, Tim K. Marks, Wenx- uan Mou, Ye Wang, Michael Jones, Anoop Cherian, Toshi- aki Koike-Akino, Xiaoming Liu, and Chen Feng. Luvli face alignment: Estimating landmarks' location, uncer- tainty, and visibility likelihood. In CVPR, 2020.","title":"Luvli face alignment: Estimating landmarks' location, uncertainty, and visibility likelihood","authors":"Abhinav Kumar, Tim K Marks, Wenxuan Mou, Ye Wang, Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, Chen Feng","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Xiaohan Li, Taotao Lai, Shuaiyu Wang, Quan Chen, Changcai Yang, Riqing Chen, Jinxun Lin, and Fu Zheng. Weighted feature pyramid networks for ob- ject detection. In ISPA/BDCloud/SocialCom/SustainCom, 2019.","title":"Weighted feature pyramid networks for object detection","authors":"Xiaohan Li, Taotao Lai, Shuaiyu Wang, Quan Chen, Changcai Yang, Riqing Chen, Jinxun Lin, Fu Zheng","year":"2019","venue":"ISPA/BDCloud/SocialCom/SustainCom","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Weijian Li, Yuhang Lu, Kang Zheng, Haofu Liao, Chihung Lin, Jiebo Luo, Chi-Tung Cheng, Jing Xiao, Le Lu, Chang-Fu Kuo, and Shun Miao. Structured landmark detection via topology-adapting deep graph learning. In ECCV, 2020.","title":"Structured landmark detection via topology-adapting deep graph learning","authors":"Weijian Li, Yuhang Lu, Kang Zheng, Haofu Liao, Chihung Lin, Jiebo Luo, Chi-Tung Cheng, Jing Xiao, Le Lu, Chang-Fu Kuo, Shun Miao","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lv et al., 2017] Jiangjing Lv, Xiaohu Shao, Junliang Xing, Cheng Cheng, and Xi Zhou. A deep regression architec- ture with two-stage re-initialization for high performance facial landmark detection. In CVPR, 2017.","title":"A deep regression architecture with two-stage re-initialization for high performance facial landmark detection","authors":"Lv","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qian et al., 2019] Shengju Qian, Keqiang Sun, Wayne Wu, Chen Qian, and Jiaya Jia. Aggregation via separation: Boosting facial landmark detector with semi-supervised style translation. In ICCV, 2019.","title":"Aggregation via separation: Boosting facial landmark detector with semi-supervised style translation","authors":"Qian","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sagonas et al., 2013] Christos Sagonas, Georgios Tz- imiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In ICCV Workshops, 2013.","title":"300 faces in-the-wild challenge: The first facial landmark localization challenge","authors":"Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, Maja Pantic","year":"2013","venue":"ICCV Workshops","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2018] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018.","title":"Integral human pose regression","authors":"Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, Yichen Wei","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tan et al., 2020] Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object de- tection. In CVPR, 2020.","title":"Efficientdet: Scalable and efficient object detection","authors":"Mingxing Tan, Ruoming Pang, V Quoc, Le","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Trigeorgis et al., 2016] George Trigeorgis, Patrick Snape, Mihalis A. Nicolaou, Epameinondas Antonakos, and Ste- fanos Zafeiriou. Mnemonic descent method: A recurrent process applied for end-to-end face alignment. In CVPR, 2016.","title":"fanos Zafeiriou. Mnemonic descent method: A recurrent process applied for end-to-end face alignment","authors":"George Trigeorgis, Patrick Snape, Mihalis A Nicolaou, Epameinondas Antonakos, Ste","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019a] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell., 2019.","title":"Deep high-resolution representation learning for visual recognition","authors":"Wang","year":"2019","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2019b] Xinyao Wang, Liefeng Bo, and Li Fuxin. Adaptive wing loss for robust face alignment via heatmap regression. In ICCV, 2019.","title":"Adaptive wing loss for robust face alignment via heatmap regression","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu and Ji, 2019] Yue Wu and Qiang Ji. Facial landmark detection: A literature survey. Int. J. Comput. Vis., 2019.","title":"Facial landmark detection: A literature survey","authors":"Yue Wu, Qiang Ji","year":"2019","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu and Yang, 2017] Wenyan Wu and Shuo Yang. Leverag- ing intra and inter-dataset variations forrobust face align- ment. In CVPRW, 2017.","title":"Leveraging intra and inter-dataset variations forrobust face alignment","authors":"Wenyan Wu, Shuo Yang","year":"2017","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2018] Wenyan (Wayne)","title":"Wenyan","authors":"Wu","year":"2018","venue":"Wenyan","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. Look at boundary: A boundary-aware face alignment algorithm. In CVPR, 2018.","title":"Look at boundary: A boundary-aware face alignment algorithm","authors":"Chen Wu, Shuo Qian, Quan Yang, Yici Wang, Qiang Cai, Zhou","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2020] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Towards explainable human pose estimation by transformer. CoRR, 2020.","title":"Transpose: Towards explainable human pose estimation by transformer","authors":"Sen Yang, Zhibin Quan, Mu Nie, Wankou Yang","year":"2020","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2020] Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, and Qianru Sun. Fea- ture pyramid transformer. In ECCV, 2020.","title":"Feature pyramid transformer","authors":"Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, Qianru Sun","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}