{"type":"ScholarlyArticle","title":"Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition","authors":[{"name":"Cheng Luo","slug":"cheng-luo","given":"Cheng","family":"Luo","affiliation":"Computer Vision Institute , Shenzhen University; Shenzhen Institute of Artificial Intelligence and Robotics for Society; Guangdong Key Laboratory of Intelligent Information Processing"},{"name":"Siyang Song","slug":"siyang-song","given":"Siyang","family":"Song","affiliation":"Department of Computer Science and Technology , University of Cambridge"},{"name":"Weicheng Xie","slug":"weicheng-xie","given":"Weicheng","family":"Xie","affiliation":"Computer Vision Institute , Shenzhen University; Shenzhen Institute of Artificial Intelligence and Robotics for Society; Guangdong Key Laboratory of Intelligent Information Processing"},{"name":"Linlin Shen","slug":"linlin-shen","given":"Linlin","family":"Shen","affiliation":"Computer Vision Institute , Shenzhen University; Shenzhen Institute of Artificial Intelligence and Robotics for Society; Guangdong Key Laboratory of Intelligent Information Processing"},{"name":"Hatice Gunes","slug":"hatice-gunes","given":"Hatice","family":"Gunes","affiliation":"Department of Computer Science and Technology , University of Cambridge"}],"abstract":"The activations of Facial Action Units (AUs) mutually influence one another. While the relationship between a pair of AUs can be complex and unique, existing approaches fail to specifically and explicitly represent such cues for each pair of AUs in each facial display. This paper proposes an AU relationship modelling approach that deep learns a unique graph to explicitly describe the relationship between each pair of AUs of the target facial display. Our approach first encodes each AU's activation status and its association with other AUs into a node feature. Then, it learns a pair of multi-dimensional edge features to describe multiple task-specific relationship cues between each pair of AUs. During both node and edge feature learning, our approach also considers the influence of the unique facial display on AUs' relationship by taking the full face representation as an input. Experimental results on BP4D and DISFA datasets show that both node and edge feature learning modules provide large performance improvements for CNN and transformer-based backbones, with our best systems achieving the state-of-the-art AU recognition results. Our approach not only has a strong capability in modelling relationship cues for AU recognition but also can be easily incorporated into various backbones. Our PyTorch code is made available at https://github.com/CVI-SZU/ME-GraphAU.","doi":"10.24963/ijcai.2022/173","url":"https://ijcai.org/proceedings/2022/173","pdf":"https://ijcai.org/proceedings/2022/0173.pdf","markdown":"https://ijcai.org/proceedings/2022/173.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/173","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":173,"pages":{"first":1239,"last":1246},"keywords":["Computer Vision: Action and Behaviour Recognition","Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Machine Learning: Multi-label","Computer Vision: Representation Learning","Machine Learning: Sequence and Graph Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986495","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984638","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bresson and Laurent, 2017] Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553, 2017.","title":"Residual gated graph convnets","authors":"Xavier Bresson, Thomas Laurent","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Churamani et al., 2021] Nikhil Churamani, Sinan Kalkan, and Hatice Gunes. Aula-caps: Lifecycle-aware capsule networks for spatio-temporal analysis of facial actions. In IEEE International Conference on Automatic Face and Gesture Recognition (FG), pages 01-08, 2021.","title":"Aula-caps: Lifecycle-aware capsule networks for spatio-temporal analysis of facial actions","authors":"Nikhil Churamani, Sinan Kalkan, Hatice Gunes","year":"2021","venue":"IEEE International Conference on Automatic Face and Gesture Recognition (FG)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Corneanu et al., 2018] Ciprian Corneanu, Meysam Madadi, and Sergio Escalera. Deep structure inference network for facial action unit recognition. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 298- 313, 2018.","title":"Deep structure inference network for facial action unit recognition","authors":"Ciprian Corneanu, Meysam Madadi, Sergio Escalera","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eleftheriadis et al., 2015] Stefanos Eleftheriadis, Ognjen Rudovic, and Maja Pantic. Multi-conditional latent vari- able model for joint facial action unit detection. In Pro- ceedings of the IEEE International Conference on Com- puter Vision (ICCV), pages 3792-3800, 2015.","title":"Multi-conditional latent variable model for joint facial action unit detection","authors":"Stefanos Eleftheriadis, Ognjen Rudovic, Maja Pantic","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Friesen and Ekman, 1978] E Friesen and Paul Ekman. Fa- cial action coding system: a technique for the measure- ment of facial movement. Palo Alto, 1978.","title":"Facial action coding system: a technique for the measurement of facial movement","authors":"Ekman ; E Friesen, Paul Friesen, Ekman","year":"1978","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gong and Cheng, 2019] Liyu Gong and Qiang Cheng. Ex- ploiting edge features for graph neural networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9211-9219, 2019.","title":"Exploiting edge features for graph neural networks","authors":"Cheng ; ] Liyu Gong, Qiang Gong, Cheng","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jacob and Stenger, 2021] Geethu Miriam Jacob and Bjorn Stenger. Facial action unit detection with transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7680-7689, 2021.","title":"Facial action unit detection with transformers","authors":"Miriam Geethu, Bjorn Jacob, Stenger","year":"2021","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2018] Wei Li, Farnaz Abtahi, Zhigang Zhu, and Lijun Yin. Eac-net: Deep nets with enhancing and crop- ping for facial action unit detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(11):2583- 2596, 2018.","title":"Eac-net: Deep nets with enhancing and cropping for facial action unit detection","authors":"Wei Li, Farnaz Abtahi, Zhigang Zhu, Lijun Yin","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019a] Guanbin Li, Xin Zhu, Yirui Zeng, Qing Wang, and Liang Lin. Semantic relationships guided rep- resentation learning for facial action unit recognition. In Proceedings of the AAAI Conference on Artificial Intelli- gence (AAAI), volume 33, pages 8594-8601, 2019.","title":"Semantic relationships guided representation learning for facial action unit recognition","authors":"Li","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019b] Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen. Self-supervised representation learning from videos for facial action unit detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10924-10933, 2019.","title":"Self-supervised representation learning from videos for facial action unit detection","authors":"Li","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2020] Zhilei Liu, Jiahui Dong, Cuicui Zhang, Longbiao Wang, and Jianwu Dang. Relation modeling with graph convolutional networks for facial action unit detection. In International Conference on Multimedia Modeling, pages 489-501, 2020.","title":"Relation modeling with graph convolutional networks for facial action unit detection","authors":"Liu","year":"2020","venue":"International Conference on Multimedia Modeling","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Martinez et al., 2017] Brais Martinez, Michel F Valstar, Bi- han Jiang, and Maja Pantic. Automatic analysis of facial actions: A survey. IEEE Transactions on Affective Com- puting, 10(3):325-347, 2017.","title":"Automatic analysis of facial actions: A survey","authors":"Brais Martinez, Michel F Valstar, Bihan Jiang, Maja Pantic","year":"2017","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mavadati et al., 2013] S Mohammad Mavadati, Moham- mad H Mahoor, Kevin Bartlett, Philip Trinh, and Jef- frey F Cohn. Disfa: A spontaneous facial action inten- sity database. IEEE Transactions on Affective Computing, 4(2):151-160, 2013.","title":"Disfa: A spontaneous facial action intensity database","authors":"Mohammad H Mohammad Mavadati, Kevin Mahoor, Philip Bartlett, Jeffrey F Trinh, Cohn","year":"2013","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Niu et al., 2019] Xuesong Niu, Hu Han, Songfan Yang, Yan Huang, and Shiguang Shan. Local relationship learning with person-specific shape regularization for facial action unit detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11917-11926, 2019.","title":"Local relationship learning with person-specific shape regularization for facial action unit detection","authors":"Xuesong Niu, Hu Han, Songfan Yang, Yan Huang, Shiguang Shan","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ridnik et al., 2021] Tal Ridnik, Emanuel Ben-Baruch, Na- dav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 82-91, 2021.","title":"Asymmetric loss for multi-label classification","authors":"Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, Lihi Zelnik-Manor","year":"2021","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shao et al., 2018] Zhiwen Shao, Zhilei Liu, Jianfei Cai, and Lizhuang Ma. Deep adaptive attention for joint facial ac- tion unit detection and face alignment. In Proceedings of the European Conference on Computer Vision (ECCV), pages 705-720, 2018.","title":"Deep adaptive attention for joint facial action unit detection and face alignment","authors":"Zhiwen Shao, Zhilei Liu, Jianfei Cai, Lizhuang Ma","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shao et al., 2019] Zhiwen Shao, Zhilei Liu, Jianfei Cai, Yunsheng Wu, and Lizhuang Ma. Facial action unit de- tection using attention and relation learning. IEEE Trans- actions on Affective Computing, 2019.","title":"Facial action unit detection using attention and relation learning","authors":"Zhiwen Shao, Zhilei Liu, Jianfei Cai, Yunsheng Wu, Lizhuang Ma","year":"2019","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shao et al., 2021a] Zhiwen Shao, Zhilei Liu, Jianfei Cai, and Lizhuang Ma. Jaa-net: joint facial action unit detec- tion and face alignment via adaptive attention. Interna- tional Journal of Computer Vision, 129(2):321-340, 2021.","title":"Jaa-net: joint facial action unit detection and face alignment via adaptive attention","authors":"Shao","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shao et al., 2021b] Zilong Shao, Siyang Song, Shashank Jaiswal, Linlin Shen, Michel Valstar, and Hatice Gunes. Personality recognition by modelling person-specific cog- nitive processes using graph representation. In Proceed- ings of the ACM International Conference on Multimedia, pages 357-366, 2021.","title":"Personality recognition by modelling person-specific cognitive processes using graph representation","authors":"Shao","year":"2021","venue":"Proceedings of the ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2021a] Siyang Song, Zilong Shao, Shashank Jaiswal, Linlin Shen, Michel Valstar, and Hatice Gunes. Learning graph representation of person-specific cognitive processes from audio-visual behaviours for automatic per- sonality recognition. arXiv preprint arXiv:2110.13570, 2021.","title":"Learning graph representation of person-specific cognitive processes from audio-visual behaviours for automatic personality recognition","authors":"Song","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2021b] Tengfei Song, Lisha Chen, Wenming Zheng, and Qiang Ji. Uncertain graph neural networks for facial action unit detection. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 1, 2021.","title":"Uncertain graph neural networks for facial action unit detection","authors":"Song","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Song et al., 2021c] Tengfei Song, Zijun Cui, Wenming Zheng, and Qiang Ji. Hybrid message passing with performance-driven structures for facial action unit detec- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6267- 6276, 2021.","title":"Hybrid message passing with performance-driven structures for facial action unit detection","authors":"Song","year":"2021","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2021] Huiyuan Yang, Lijun Yin, Yi Zhou, and Jiuxiang Gu. Exploiting semantic embedding and visual feature for facial action unit detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10482-10491, 2021.","title":"Exploiting semantic embedding and visual feature for facial action unit detection","authors":"Huiyuan Yang, Lijun Yin, Yi Zhou, Jiuxiang Gu","year":"2021","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yin and Liu, 2017] Xi Yin and Xiaoming Liu. Multi- task convolutional neural network for pose-invariant face recognition. IEEE Transactions on Image Processing, 27(2):964-975, 2017.","title":"Multitask convolutional neural network for pose-invariant face recognition","authors":"Yin, Liu ; ] Xi Yin, Xiaoming Liu","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2014] Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, Peng Liu, and Jeffrey M Girard. Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database. Image and Vision Computing, 32(10):692-706, 2014.","title":"Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database","authors":"Xing Zhang, Lijun Yin, Shaun Jeffrey F Cohn, Michael Canavan, Andy Reale, Peng Horowitz, Jeffrey M Liu, Girard","year":"2014","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2016] Kaili Zhao, Wen-Sheng Chu, and Hong- gang Zhang. Deep region and multi-label learning for facial action unit detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3391-3399, 2016.","title":"Deep region and multi-label learning for facial action unit detection","authors":"Kaili Zhao, Wen-Sheng Chu, Honggang Zhang","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}