{"type":"ScholarlyArticle","title":"MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing","authors":[{"name":"Ajian Liu","slug":"ajian-liu","given":"Ajian","family":"Liu","affiliation":"School of Computer Science and Engineering , Faculty of Innovation Engineering , Macau University of Science and Technology , Macau; CBSR&NLPR , Institute of Automation , Chinese Academy of Sciences , Beijing , China"},{"name":"Yanyan Liang","slug":"yanyan-liang","given":"Yanyan","family":"Liang","affiliation":"School of Computer Science and Engineering , Faculty of Innovation Engineering , Macau University of Science and Technology , Macau"}],"abstract":"The existing multi-modal face anti-spoofing (FAS) frameworks are designed based on two strategies: halfway and late fusion. However, the former requires test modalities consistent with the training input, which seriously limits its deployment scenarios. And the latter is built on multiple branches to process different modalities independently, which limits their use in applications with low memory or fast execution requirements. In this work, we present a single branch based Transformer framework, namely Modality-Agnostic Vision Transformer (MA-ViT), which aims to improve the performance of arbitrary modal attacks with the help of multi-modal data. Specifically, MA-ViT adopts the early fusion to aggregate all the available training modalities’ data and enables flexible testing of any given modal samples. Further, we develop the Modality-Agnostic Transformer Block (MATB) in MA-ViT, which consists of two stacked attentions named Modal-Disentangle Attention (MDA) and Cross-Modal Attention (CMA), to eliminate modality-related information for each modal sequences and supplement modality-agnostic liveness features from another modal sequences, respectively. Experiments demonstrate that the single model trained based on MA-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin, and approaches the multi-modal frameworks introduced with smaller FLOPs and model parameters.","doi":"10.24963/ijcai.2022/165","url":"https://ijcai.org/proceedings/2022/165","pdf":"https://ijcai.org/proceedings/2022/0165.pdf","markdown":"https://ijcai.org/proceedings/2022/165.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/165","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":165,"pages":{"first":1180,"last":1186},"keywords":["Computer Vision: Biometrics, Face, Gesture and Pose Recognition"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986872","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985015","length_sec":326,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Boulkenafet et al., 2017] Zinelabinde Boulkenafet, Jukka Komulainen, Lei Li, Xiaoyi Feng, and Abdenour Hadid. Oulu-npu: A mobile face presentation attack database with real-world variations. In FGR, pages 612-618, 2017.","title":"Oulu-npu: A mobile face presentation attack database with real-world variations","authors":"Boulkenafet","year":"2017","venue":"FGR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[George and Marcel, 2019] Anjith George and Sébastien Marcel. Deep pixel-wise binary supervision for face pre- sentation attack detection. In ICB, pages 1-8, 2019.","title":"Anjith George and Sébastien Marcel. Deep pixel-wise binary supervision for face presentation attack detection","authors":"Marcel George","year":"2019","venue":"ICB","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[George and Marcel, 2020] Anjith George and Sébastien Marcel. Learning one class representations for face pre- sentation attack detection using multi-channel convolu- tional neural networks. IEEE TIFS, 16:361-375, 2020.","title":"Anjith George and Sébastien Marcel. Learning one class representations for face presentation attack detection using multi-channel convolutional neural networks","authors":"Marcel George","year":"2020","venue":"IEEE TIFS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[George and Marcel, 2021] Anjith George and Sébastien Marcel. Cross modal focal loss for rgbd face anti-spoofing. In CVPR, pages 7882-7891, 2021.","title":"Anjith George and Sébastien Marcel. Cross modal focal loss for rgbd face anti-spoofing","authors":"Marcel George","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[George et al., 2019] Anjith George, Zohreh Mostaani, David Geissenbuhler, Olegs Nikisins, André Anjos, and Sébastien Marcel. Biometric face presentation attack de- tection with multi-channel convolutional neural network. IEEE TIFS, 15:42-55, 2019.","title":"Biometric face presentation attack detection with multi-channel convolutional neural network","authors":"George","year":"2019","venue":"IEEE TIFS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Liu et al., 2016] Siqi Liu, Baoyao Yang, Pong C Yuen, and Guoying Zhao. A 3d mask face anti-spoofing database with real world variations. In CVPRW, pages 100-106, 2016.","title":"A 3d mask face anti-spoofing database with real world variations","authors":"Siqi Liu, Baoyao Yang, Pong C Yuen, Guoying Zhao","year":"2016","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2018] Yaojie Liu, Amin Jourabloo, and Xiaom- ing Liu. Learning deep models for face anti-spoofing: Bi- nary or auxiliary supervision. In CVPR, pages 389-398, 2018.","title":"Learning deep models for face anti-spoofing: Binary or auxiliary supervision","authors":"Liu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2019] Ajian Liu, Jun Wan, Sergio Escalera, Hugo Jair Escalante, Zichang Tan, Qi Yuan, Kai Wang, Chi Lin, Guodong Guo, Isabelle Guyon, et al. Multi-modal face anti-spoofing attack detection challenge at cvpr2019. In CVPRW, pages 0-0, 2019.","title":"Multi-modal face anti-spoofing attack detection challenge at cvpr2019","authors":"Liu","year":"2019","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2020] Yaojie Liu, Joel Stehouwer, and Xiaoming Liu. On disentangling spoof trace for generic face anti- spoofing. In ECCV, pages 406-422, 2020.","title":"On disentangling spoof trace for generic face antispoofing","authors":"Liu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021a] Ajian Liu, Xuan Li, Jun Wan, Yanyan Liang, Sergio Escalera, Hugo Jair Escalante, Meysam Madadi, Yi Jin, Zhuoyuan Wu, Xiaogang Yu, et al. Cross- ethnicity face anti-spoofing recognition challenge: A re- view. IET Biometrics, 10(1):24-43, 2021.","title":"Crossethnicity face anti-spoofing recognition challenge: A review","authors":"Liu","year":"2021","venue":"IET Biometrics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2021b] Ajian Liu, Zichang Tan, Jun Wan, Ser- gio Escalera, Guodong Guo, and Stan Z Li. Casia-surf cefa: A benchmark for multi-modal cross-ethnicity face anti-spoofing. In WACV, pages 1179-1187, 2021.","title":"Casia-surf cefa: A benchmark for multi-modal cross-ethnicity face anti-spoofing","authors":"Liu","year":"2021","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021c] Ajian Liu, Zichang Tan, Jun Wan, Yanyan Liang, Zhen Lei, Guodong Guo, and Stan Z Li. Face anti- spoofing via adversarial cross-modality translation. IEEE TIFS, 16:2759-2772, 2021.","title":"Face antispoofing via adversarial cross-modality translation","authors":"Liu","year":"2021","venue":"IEEE TIFS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021d] Ajian Liu, Chenxu Zhao, Zitong Yu, Anyang Su, Xing Liu, Zijian Kong, Jun Wan, Sergio Es- calera, Hugo Jair Escalante, Zhen Lei, et al. 3d high- fidelity mask face presentation attack detection challenge. In ICCVW, pages 814-823, 2021.","title":"3d highfidelity mask face presentation attack detection challenge","authors":"Liu","year":"2021","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021e] Ajian Liu, Chenxu Zhao, Zitong Yu, Jun Wan, Anyang Su, Xing Liu, Zichang Tan, Sergio Escalera, Junliang Xing, Yanyan Liang, et al. Contrastive context- aware learning for 3d high-fidelity mask face presentation attack detection. arXiv preprint arXiv:2104.06148, 2021.","title":"Contrastive contextaware learning for 3d high-fidelity mask face presentation attack detection","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021f] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parkin and Grinchuk, 2019] Aleksandr Parkin and Oleg Grinchuk. Recognizing multi-modal face spoofing with face recognition networks. In CVPRW, pages 0-0, 2019.","title":"Recognizing multi-modal face spoofing with face recognition networks","authors":"Aleksandr Parkin, Oleg Grinchuk","year":"2019","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shao et al., 2019] Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-adversarial discriminative deep do- main generalization for face presentation attack detection. In CVPR, pages 10023-10031, 2019.","title":"Multi-adversarial discriminative deep domain generalization for face presentation attack detection","authors":"Rui Shao, Xiangyuan Lan, Jiawei Li, Pong C Yuen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pages 10347-10357, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021] Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Wen Xie, Hao Li, and Rong Jin. Kvt: k-nn attention for boosting vision transformers. arXiv preprint arXiv:2106.00515, 2021.","title":"Kvt: k-nn attention for boosting vision transformers","authors":"Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Wen Xie, Hao Li, Rong Jin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Xu et al., ] Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, and Rong Jin. Cdtrans: Cross-domain transformer for unsupervised domain adaptation. arXiv preprint arXiv:2109.06165.","title":"Cdtrans: Cross-domain transformer for unsupervised domain adaptation","authors":"Xu","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2020] Zitong Yu, Chenxu Zhao, Zezheng Wang, Yunxiao Qin, Zhuo Su, Xiaobai Li, Feng Zhou, and Guoy- ing Zhao. Searching central difference convolutional net- works for face anti-spoofing. In CVPR, pages 5295-5305, 2020.","title":"Searching central difference convolutional networks for face anti-spoofing","authors":"Yu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2019] Shifeng Zhang, Xiaobo Wang, Ajian Liu, Chenxu Zhao, Jun Wan, Sergio Escalera, Hailin Shi, Zezheng Wang, and Stan Z Li. A dataset and benchmark for large-scale multi-modal face anti-spoofing. In CVPR, pages 919-928, 2019.","title":"A dataset and benchmark for large-scale multi-modal face anti-spoofing","authors":"Shifeng Zhang, Xiaobo Wang, Ajian Liu, Chenxu Zhao, Jun Wan, Sergio Escalera, Hailin Shi, Zezheng Wang, Stan Z Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2020a] Ke-Yue Zhang, Taiping Yao, Jian Zhang, Ying Tai, Shouhong Ding, Jilin Li, Feiyue Huang, Haichuan Song, and Lizhuang Ma. Face anti-spoofing via disentangled representation learning. In ECCV, pages 641-657, 2020.","title":"Face anti-spoofing via disentangled representation learning","authors":"Zhang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2020b] Shifeng Zhang, Ajian Liu, Jun Wan, Yanyan Liang, Guodong Guo, Sergio Escalera, Hugo Jair Escalante, and Stan Z Li. Casia-surf: A large-scale multi- modal benchmark for face anti-spoofing. IEEE TBIOM, 2(2):182-193, 2020.","title":"Casia-surf: A large-scale multimodal benchmark for face anti-spoofing","authors":"Zhang","year":"2020","venue":"IEEE TBIOM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}