{"type":"ScholarlyArticle","title":"1DFormer: A Transformer Architecture Learning 1D Landmark Representations for Facial Landmark Tracking","authors":[{"name":"Shi Yin","slug":"shi-yin","given":"Shi","family":"Yin","affiliation":"iFLYTEK Research; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center"},{"name":"Shijie Huang","slug":"shijie-huang","given":"Shijie","family":"Huang","affiliation":"iFLYTEK Research"},{"name":"Shangfei Wang","slug":"shangfei-wang","given":"Shangfei","family":"Wang","affiliation":"University of Science and Technology of China"},{"name":"Jinshui Hu","slug":"jinshui-hu","given":"Jinshui","family":"Hu","affiliation":"iFLYTEK Research"},{"name":"Tao Guo","slug":"tao-guo","given":"Tao","family":"Guo","affiliation":"iFLYTEK Research"},{"name":"Bing Yin","slug":"bing-yin","given":"Bing","family":"Yin","affiliation":"iFLYTEK Research"},{"name":"Baocai Yin","slug":"baocai-yin","given":"Baocai","family":"Yin","affiliation":"iFLYTEK Research"},{"name":"Cong Liu","slug":"cong-liu","given":"Cong","family":"Liu","affiliation":"iFLYTEK Research"}],"abstract":"Recently, heatmap regression methods based on 1D landmark representations have shown prominent performance on locating facial landmarks.  However, previous  methods ignored to make deep explorations on the good potentials of 1D landmark representations for sequential and structural modeling of multiple landmarks to track facial landmarks. To address this limitation, we propose a Transformer architecture, namely 1DFormer, which learns informative 1D landmark representations by capturing the dynamic and the geometric patterns of landmarks via token communications in both temporal and spatial dimensions for facial landmark tracking. For temporal modeling, we propose a confidence-enhanced multi-head attention mechanism with a recurrently token mixing strategy to adaptively and robustly embed long-term landmark dynamics into their 1D representations; for structure modeling, we design intra-group and inter-group geometric encoding mechanisms to encode the component-level as well as global-level facial structure patterns as a refinement for the 1D representations of landmarks through token communications in the spatial dimension via 1D convolutional layers. Experimental results on the 300VW and the TF databases show that 1DFormer successfully models the long-range sequential patterns as well as the inherent facial structures to learn informative 1D representations of landmark sequences, and achieves state-of-the-art performance on facial landmark tracking. Codes of our model are available in the supplementary materials.","doi":"10.24963/ijcai.2024/176","url":"https://ijcai.org/proceedings/2024/176","pdf":"https://ijcai.org/proceedings/2024/0176.pdf","markdown":"https://ijcai.org/proceedings/2024/176.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/176","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":176,"pages":{"first":1588,"last":1597},"keywords":["Computer Vision: CV: Representation learning","Computer Vision: CV: Motion and tracking","Computer Vision: CV: Biometrics, face, gesture and pose recognition"],"videos":[],"references":[{"position":1,"raw":"[Bulat and Tzimiropoulos, 2017] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In ICCV, pages 1021-1030, 2017.","title":"How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)","authors":"Tzimiropoulos Bulat, Adrian Bulat, Georgios Tzimiropoulos","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chandran et al., 2020] Prashanth Chandran, Derek Bradley, Markus H. Gross, and Thabo Beeler. Attention-driven cropping for very high resolution facial landmark detec- tion. In CVPR, pages 5860-5869, 2020.","title":"Attention-driven cropping for very high resolution facial landmark detection","authors":"Prashanth Chandran, Derek Bradley, Markus H Gross, Thabo Beeler","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chi et al., 2022] Cailong Chi, Dong Zhang, Zhesi Zhu, Xingzhi Wang, and Dah-Jye Lee. Human pose estimation for low-resolution image using 1-d heatmaps and offset re- gression. Multimedia Tools and Applications, pages 1-19, 2022.","title":"Human pose estimation for low-resolution image using 1-d heatmaps and offset regression","authors":"Cailong Chi, Dong Zhang, Zhesi Zhu, Xingzhi Wang, Dah-Jye Lee","year":"2022","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chrysos et al., 2018] Grigorios G Chrysos, Epameinondas Antonakos, Patrick Snape, Akshay Asthana, and Stefanos Zafeiriou. A comprehensive performance evaluation of de- formable face tracking \"in-the-wild\". IJCV, 126(2-4):198- 232, 2018.","title":"A comprehensive performance evaluation of deformable face tracking \"in-the-wild","authors":"Epameinondas Grigorios G Chrysos, Patrick Antonakos, Akshay Snape, Stefanos Asthana, Zafeiriou","year":"2018","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[FGNET, 2014] FGNET. Talking face video. http: //www-prima.inrialpes.fr/FGnet/data/01-TalkingFace/ talking face.html, 2014.","title":"FGNET. Talking face video","authors":"Fgnet","year":"2014","venue":null,"doi":null,"url":"http://www-prima.inrialpes.fr/FGnet/data/01-TalkingFace/talkingface.html","internal_url":null},{"position":6,"raw":"[Kang et al., 2023] Seoungyoon Kang, Minhyun Lee, Min- jae Kim, and Hyunjung Shim. Hybridmatch: Semi- supervised facial landmark detection via hybrid heatmap representations. IEEE Access, 11:26125-26135, 2023.","title":"Hybridmatch: Semisupervised facial landmark detection via hybrid heatmap representations","authors":"Kang","year":"2023","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2013] Y. Li, S. Wang, Y. Zhao, and Q. Ji. Simulta- neous facial feature tracking and facial expression recog- nition. TIP, 22(7):2559-2573, 2013.","title":"Simultaneous facial feature tracking and facial expression recognition","authors":"Y Li, S Wang, Y Zhao, Q Ji","year":"2013","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2021] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log-likelihood estimation. In ICCV, pages 11005-11014, 2021.","title":"Human pose regression with residual log-likelihood estimation","authors":"Li","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2022a] Hui Li, Zidong Guo, Seon-Min Rhee, Se- ungju Han, and Jae-Joon Han. Towards accurate facial landmark detection via cascaded transformers. In CVPR, pages 4166-4175, 2022.","title":"Towards accurate facial landmark detection via cascaded transformers","authors":"Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2022b] Jinpeng Li, Haibo Jin, Shengcai Liao, Ling Shao, and Pheng-Ann Heng. Repformer: Refinement pyramid transformer for robust facial landmark detection. In IJCAI, pages 1088-1094, 2022.","title":"Repformer: Refinement pyramid transformer for robust facial landmark detection","authors":"Li","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/152"},{"position":11,"raw":"[Li et al., 2022c] Yanjie Li, Sen Yang, Peidong Liu, Shoukui Zhang, Yunxiao Wang, Zhicheng Wang, Wankou Yang, and Shu-Tao Xia. Simcc: A simple coordinate classifi- cation perspective for human pose estimation. In ECCV, pages 89-106, 2022.","title":"Simcc: A simple coordinate classification perspective for human pose estimation","authors":"Li","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018] Hao Liu, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Two-stream transformer networks for video-based face alignment. TPAMI, 40(11):2546-2554, 2018.","title":"Two-stream transformer networks for video-based face alignment","authors":"Hao Liu, Jiwen Lu, Jianjiang Feng, Jie Zhou","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2022] Shengli Liu, Xiaowen Zhu, Zewei Cao, and Gang Wang. Deep 1d landmark representation learn- ing for space target pose estimation. Remote Sensing, 14(16):4035, 2022.","title":"Deep 1d landmark representation learning for space target pose estimation","authors":"Shengli Liu, Xiaowen Zhu, Zewei Cao, Gang Wang","year":"2022","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Moon and Lee, 2020] Gyeongsik Moon and Kyoung Mu Lee. I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a sin- gle RGB image. In ECCV, Lecture Notes in Computer Science, pages 752-768, 2020.","title":"Gyeongsik Moon and Kyoung Mu Lee. I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single RGB image","authors":"Lee Moon","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Newell et al., 2016] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose esti- mation. In ECCV, pages 483-499, 2016.","title":"Stacked hourglass networks for human pose estimation","authors":"Alejandro Newell, Kaiyu Yang, Jia Deng","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peng et al., 2016] Xi Peng, Rogerio S Feris, Xiaoyu Wang, and Dimitris N Metaxas. A recurrent encoder-decoder net- work for sequential face alignment. In ECCV, pages 38- 56, 2016.","title":"A recurrent encoder-decoder network for sequential face alignment","authors":"Xi Peng, S Rogerio, Xiaoyu Feris, Dimitris N Wang, Metaxas","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sagonas et al., 2016] Christos Sagonas, Epameinondas An- tonakos, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: Database and results. Image and vision computing, 47:3-18, 2016.","title":"Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: Database and results","authors":"Sagonas","year":"2016","venue":"Image and vision computing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shen et al., 2015] Jie Shen, Stefanos Zafeiriou, Grigoris G Chrysos, Jean Kossaifi, Georgios Tzimiropoulos, and Maja Pantic. The first facial landmark tracking in-the-wild challenge: Benchmark and results. In ICCV Workshops, pages 50-58, 2015.","title":"The first facial landmark tracking in-the-wild challenge: Benchmark and results","authors":"Jie Shen, Stefanos Zafeiriou, Jean Grigoris G Chrysos, Georgios Kossaifi, Maja Tzimiropoulos, Pantic","year":"2015","venue":"ICCV Workshops","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tai et al., 2019] Ying Tai, Yicong Liang, Xiaoming Liu, Lei Duan, Jilin Li, Chengjie Wang, Feiyue Huang, and Yu Chen. Towards highly accurate and stable face align- ment for high-resolution videos. In AAAI, pages 8893- 8900, 2019.","title":"Towards highly accurate and stable face alignment for high-resolution videos","authors":"Ying Tai, Yicong Liang, Xiaoming Liu, Lei Duan, Jilin Li, Chengjie Wang, Feiyue Huang, Yu Chen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wan et al., 2021] Jun Wan, Hui Xi, Jie Zhou, Zhihui Lai, Witold Pedrycz, Xu Wang, and Hang Sun. Robust and pre- cise facial landmark detection by self-calibrated pose at- tention network. IEEE Transactions on Cybernetics, pages 1-15, 2021.","title":"Robust and precise facial landmark detection by self-calibrated pose attention network","authors":"Jun Wan, Hui Xi, Jie Zhou, Zhihui Lai, Witold Pedrycz, Xu Wang, Hang Sun","year":"2021","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021a] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learn- ing for visual recognition. TPAMI, 43(10):3349-3364, 2021.","title":"Deep high-resolution representation learning for visual recognition","authors":"Wang","year":"2021","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2021b] Qingzhong Wang, Pengfei Zhang, Haoyi Xiong, and Jian Zhao. Face. evolve: A high- performance face recognition library. arXiv preprint arXiv:2107.08621, 2021.","title":"Face. evolve: A highperformance face recognition library","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu and Ji, 2019] Yue Wu and Qiang Ji. Facial landmark de- tection: A literature survey. IJCV, 127(2):115-142, 2019.","title":"Facial landmark detection: A literature survey","authors":"Yue Wu, Qiang Ji","year":"2019","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2014] Yue Wu, Ziheng Wang, and Qiang Ji. A hi- erarchical probabilistic model for facial feature detection. In CVPR, pages 1781-1788, 2014.","title":"A hierarchical probabilistic model for facial feature detection","authors":"Yue Wu, Ziheng Wang, Qiang Ji","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiong and De la Torre, 2013] Xuehan Xiong and Fernando De la Torre. Supervised descent method and its applica- tions to face alignment. In CVPR, pages 532-539, 2013.","title":"Xuehan Xiong and Fernando De la Torre. Supervised descent method and its applications to face alignment","authors":"De Xiong, Torre La","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiong et al., 2020] Yilin Xiong, Zijian Zhou, Yuhao Dou, and Zhizhong Su. Gaussian vector: An efficient solution for facial landmark detection. In ACCV, volume 12626, pages 70-87, 2020.","title":"Gaussian vector: An efficient solution for facial landmark detection","authors":"Yilin Xiong, Zijian Zhou, Yuhao Dou, Zhizhong Su","year":"2020","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yin et al., 2019] Shi Yin, Shangfei Wang, Guozhu Peng, Xiaoping Chen, and Bowen Pan. Capturing spatial and temporal patterns for facial landmark tracking through ad- versarial learning. In IJCAI, pages 1010-1017, 2019.","title":"Capturing spatial and temporal patterns for facial landmark tracking through adversarial learning","authors":"Shangfei Shi Yin, Guozhu Wang, Xiaoping Peng, Bowen Chen, Pan","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/142"},{"position":29,"raw":"[Yin et al., 2020] Shi Yin, Shangfei Wang, Xiaoping Chen, Enhong Chen, and Cong Liang. Attentive one-dimensional heatmap regression for facial landmark detection and tracking. In ACM MM, pages 538-546, 2020.","title":"Attentive one-dimensional heatmap regression for facial landmark detection and tracking","authors":"Shangfei Shi Yin, Xiaoping Wang, Enhong Chen, Cong Chen, Liang","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2021] Congcong Zhu, Xiaoqiang Li, Jide Li, and Songmin Dai. Improving robustness of facial landmark de- tection by defending against adversarial attacks. In ICCV, pages 11731-11740, 2021.","title":"Improving robustness of facial landmark detection by defending against adversarial attacks","authors":"Congcong Zhu, Xiaoqiang Li, Jide Li, Songmin Dai","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2022a] Congcong Zhu, Xiaoqiang Li, Jide Li, Songmin Dai, and Weiqin Tong. Reasoning structural rela- tion for occlusion-robust facial landmark localization. Pat- tern Recognition, 122:108325, 2022.","title":"Reasoning structural relation for occlusion-robust facial landmark localization","authors":"Zhu","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2022b] Congong Zhu, Xiaoqiang Li, Jide Li, Songmin Dai, and Weiqin Tong. Multi-sourced knowl- edge integration for robust self-supervised facial landmark tracking. IEEE Transactions on Multimedia, pages 1-13, 2022.","title":"Multi-sourced knowledge integration for robust self-supervised facial landmark tracking","authors":"Zhu","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}