Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition

1 Introduction

Facial Action Coding System [Friesen and Ekman, 1978] represents human face by a set of facial muscle movements called Action Units (AUs). Compared with the emotionbased categorical facial expression model, AUs describe human facial expressions in a more comprehensive and objective manner [Martinez et al., 2017]. Facial AU recognition is a multi-label classification problem as multiple AUs can be activated simultaneously. While previous studies found that underlying relationships among AUs' activation [Corneanu et al., 2018; Song et al., 2021c; Shao et al., 2021a] are crucial for their recognition, how to properly model such relationships is still an open research question in the field.

∗Equal contribution.

†Corresponding author.

A popular strategy applies various traditional machine learning models (e.g., conditional models [Eleftheriadis et al., 2015]) or neural network-based operations (e.g., convolution [Zhao et al., 2016], Long-Short-Term-Memory networks [Niu et al., 2019] or attention [Shao et al., 2021a]), to encode all AU descriptors as a single representation which reflects the underlying relationship among all AUs. A key drawback of such solutions is that they fail to individually model the relationship between each pair of AUs, which may contain crucial cues for their recognition (Problem 1). Some studies represent all AUs of the target face as a graph, where each AU is represented as a node, and each pair of AUs relationship is specifically described by an edge that contains a binary value or a single weight to describe their connectivity or association [Song et al., 2021b; Song et al., 2021c]. However, a single value may not be enough to represent the complex underlying relationship between a pair of AUs (Problem 2). In particular, some studies [Li et al., 2019a; Liu et al., 2020] even manually define a single graph topology for all face images based on prior knowledge (e.g., AUs co-occurrence pattern), which fails to consider the influences of the unique facial display on AU relationships (Problem 3).

In this paper, we propose a novel AUs relationship modelling approach to address the problems described above, which can be easily incorporated with various deep learning backbones. It takes a full face representation produced by the backbone as the input, and outputs an AUs relation graph that explicitly describes the relationship between each pair of AUs (addressing problem 1). Specifically, our approach consists of two modules: (i) the AUs relationship-aware node feature learning (ANFL) module first individually learns a representation for each AU from the input full face representation (Sec. 2.1), which encodes not only the AU's activation status but also its association with other AUs; and then (ii) the multi-dimensional edge feature learning (MEFL) module learns multiple task-specific relationship cues as the edge representation for each pair of AUs (Sec. 2.2) (addressing problem 2). Since both node and edge feature learning take the full face representation as the input, the influence of the unique facial display on AUs relationship is considered when generating its AUs relation graph (addressing problem 3).

In summary, the main contributions of our AU relationship modelling approach are that it represents AU relationships as a unique graph for each facial display, which (i) encodes both the activation status of the AU and its association with other AUs into each node feature; and (ii) learns a multi-dimensional edge feature to explicitly capture the taskspecific relationship cues between each pair of AUs. Our multi-dimensional edge encodes unique and multiple relationships between each pair of AUs, rather than a single relationship (e.g., spatial adjacency, co-occurrence patterns, etc.) that the single value-edge encoded, which would theoretically generalizes better in modeling complex relationships between vertices [Gong and Cheng, 2019; Song et al., 2021a; Shao et al., 2021b]. The main novelty of the proposed approach in comparison to pre-defined AU graphs [Li et al., 2019a; Liu et al., 2020] and deep learned facial displayspecific graphs [Song et al., 2021b; Song et al., 2021c] are illustrated in Figure 1. To the best of our knowledge, this is the first CNN-GCN approach that conducts end-to-end multidimensional edge feature learning for face image processing tasks. The pipeline of the proposed approach is illustrated in Figure 2.

2 The Proposed Approach

Our AU relationship modelling approach deep learns a unique AU relation graph from the representation of the target face, which explicitly captures recognition-related relationship cues among AUs based on the end-to-end learned relationship modelling modules. The learned AU relation graph represents the ith AU as the node vi ∈ V in the graph, which contains a vector describing the activation status of the ith AU as well as its association with other AUs in the target facial display. Besides, the task-specific relationship cues between nodes (AUs) vi and vj are also explicitly described by two directed edges ei,j, ej,i ∈ E that are represented by two deep learned vectors.

2.1 AUs Relationship-aware Node Feature Learning

As illustrated in Figure 2, the ANFL module consists of two blocks: an AU-specific Feature Generator (AFG) and a Facial Graph Generator (FGG). The AFG individually generates a representation for each AU, based on which the FGG automatically designs an optimal graph for each facial display, aiming to accurately recognize all target AUs. To achieve this, the FGG would enforce the AFG to encode task-specific associations among AUs into their AU-specific representations.

AU-specific Feature Generator

The AFG is made up of N AU-specific feature extractors, each of which contains a fully connected layer (FC) and a global average pooling (GAP) layer. It takes the full face representation X ∈ RD×C (C channels with D dimensions) as the input, which can be produced by any standard machine learning backbone. The FC layer of ith AU-specific feature extractor first projects the X to an AU-specific feature map Ui ∈ RD×C , which is then fed to a GAP layer, yielding a vector containing C values as the ith AU's representation vi. Consequently, N AU representations can be learned from the full face representationX, respectively.

Facial Graph Generator

Our hypothesis is that the relationship cues among AUs are unique for each facial display. As a result, directly utilizing relationship cues defined in the training set (e.g., cooccurrence pattern) may not generalise well at the inference stage. As a result, we propose to represent AU relationships in each facial display as a unique graph which considers the influence of the target facial display on AUs relationship.

For a face image, the FGG block treats N target AUs' feature vectors V = {v1, v2, · · · , vN} as N node features and defines the connectivity (edge presence) between a pair of nodes vi and vj by their features' similarity (Sim(i, j) = vT i vj). Specifically, we choose the K nearest neighbours of each node as its neighbours, and thus the graph topology is defined by the learned node features. Then, a GCN layer is employed to jointly update all AUs activation status from the produced graph, where the ith AU's activation representation vFGG i is generated by vi and its connected nodes as:

where σ[] is the non-linear activation; g and r denote differentiable functions of the GCN layer, and ai,j ∈ {0, 1} represents the connectivity between vi and vj .

To provide a prediction for the ith AU, we propose a similarity calculating (SC) strategy which learns a trainable vector si that has the same dimension as the vFGG i , and then generates the ith AU's occurrence probability by computing the cosine similarity between vFGG i and si as:

where ReLU denotes a non-linearity activation. As a result, a pair of AUs that have a strong association (high similarity) would have connected nodes. In other words, the FGG block enforces the AFG block to encode node (AU) features that contain task-specific relationship cues among AUs of the target facial display, in order to produce an optimal graph for their recognition.

2.2 Multi-dimensional Edge Feature Learning

In addition to relationship cues encoded in node features, we also propose a Multi-dimensional Edge Feature Learning (MEFL) module to deep learn a pair of multi-dimensional edge features, aiming to explicitly describe task-specific relationship cues between each pair of AUs. Importantly, we learn edge features for both connected and un-connected node pairs defined in Sec. 2.1. Even when a pair of nodes have low similarity, their relationship may still contain crucial cues for AU recognition, which are ignored during the node feature learning. Since an AU's activation may also influence other AUs' status, the relationship between a pair of AUs can be reflected by not only their features but also AUs defined by other facial regions. Thus, the MEFL module consists of two blocks: a Facial display-specific AU representation modelling (FAM) block that first locates activation cues of each AU from the full face representation, and an AU relationship modelling (ARM) block that further extracts features from these located cues, which relate to both AUs activation. This is also illustrated in Figure 3.

FAM. As illustrated in Figure 3, for a pair of AUs, the FAM takes their AU-specific feature maps Ui, Uj , and the full face representationX as the input. It first conducts cross attention between Ui and X as well as Uj and X, respectively, where AU-specific feature maps Ui and Uj are individually used as queries, while the full face representation X is treated as the key and value. This process can be formulated as:

with the cross attention operation in FAM defined as:

where Wq, Wk and Wv are learnable weights, and dk is a scaling factor equalling to the number of the 'key's' channels.

As a result, the produced F AS i,x and F AS j,x extract and highlight the most important facial cues from all facial regions of the target facial display for AU i and AU j's recognition, respectively, which consider the influence of the unique facial display on AUs relationships.

ARM. After encoding task-specific facial cues for each AU's recognition independently, the ARM block further extracts the facial cues related to both AUs' recognition. It also conducts the cross-attention (has the same form as Eq. 4 but independent weights) between F AS i,x and F AS j,x , and produces features F AR i,j,x and F AR j,i,x , where F AR i,j,x is generated by using F AS j,x as the query and F AS i,x as the key and value, while F AR j,i,x is generated by using F AS i,x as the query and F AS j,x as the key and value. As a result, the F AR i,j,x summarizes F AS j,x -related cues in the F AS i,x , and F AR j,i,x summarizes F AS i,x -related cues in the F AS j,x . Finally, we feed F AR i,j,x and F AR j,i,x to a GAP layer to obtain multi-dimensional edge feature vectors ei,j and ej,i, respectively. Mathematically speaking, this process can be represented as

In short, the features encoded in edge features ei,j and ej,i summarize multiple facial cues that relate to both ith and jth AUs' recognition, from all facial regions of the target face.

Once the AUs relation graph G0 = (V 0 ,E0 ) that consists of N node features and N × N multi-dimensional directed edge features is learned, we feed it to a GCN model to jointly recognize all target AUs. In this paper, we use a model that only consists of L gated graph convolution layers (GatedGCN) [Bresson and Laurent, 2017], and thus the output GL = (V L,EL) is also a graph that has the same topology as G0 , where the ith node vL i represents the ith AU's activation status (L = 2 in this paper). We finally re-employ the SC module proposed in the FGG block to predict N AUs' activation from the node features of GL. During the inference stage, only the well-trained AFG and MEFL are used to process the input full face representation and generate the AU relation graph.

2.3 Training Strategy

In this paper, we propose a two-stage training method to jointly optimize the proposed ANFL and MEFL modules with the backbone and classifier in an end-to-end manner.

In the first stage, we train the backbone with the ANFL module, aiming to learn an AFG block that produces node features containing both AU activation status and their associations for each facial display. A priori, we notice that existing AU datasets usually have imbalanced labels, where some AUs occurred less frequently than others, and most AUs are inactivated for the majority of face images. To alleviate such issues, we propose a weighted asymmetric loss to compute the loss value between the ground-truth and predictions generated by the FGG block. It is inspired by the asymmetric loss [Ridnik et al., 2021], but has a unique weight for each subtask (each AU's recognition) as well as fewer hyperparameters. The proposed weighted asymmetric loss is formulated as:

where pi, yi and wi are the prediction (occurrence probability), ground truth and weight of the ith AU, respectively. Here, the wi = N(1/ri)/ΣNj=1 (1/rj) is defined by the ith AU's occurrence rate ri computed from the training set. It allows loss values to account less for AUs that have higher occurrence rates in the training set, leading loss values caused by less frequently occurring AUs to have higher weights during the training. Additionally, the term 'pi' in the center of (1 - yi)pilog(1 - pi) down weights loss values caused by inactivated AUs that are easy to be recognized, whose predicted occurrence probabilities are close to zero (pi ≪ 0.5), enforcing the training process to focus on activated AUs and inactivated AUs that are hard to be correctly recognized.

The second stage trains the MEFL module and classifier (GatedGCN) with the pre-trained backbone and AFG block. Here, we again employ the proposed weighted asymmetric loss (Eq. 6) to compute the loss value LWA between the outputs of the classifier and ground truth. Additionally, we also leverage the AUs co-occurrence patterns to supervise the training process. We feed multi-dimensional edge features eLi,j and eLj,i generated from the last GatedGCN layer to a shared FC layer, in order to predict the co-occurrence pattern of the ith and jth AUs of the target face. We define this task as a four-class classification problem, i.e., for a pair of nodes vi and vj: (1) both vi and vj are inactivated; (2) vi is inactivated and vj is activated; (3) vi is activated and vj is inactivated; or (4) both vi and vj are activated. As a result, the categorical cross-entropy loss is introduced as:

where |E| denotes the number of edges in the facial graph; NE is the number of co-occurrence patterns; pei,j is the cooccurrence prediction output from the shared FC layer. Consequently, The overall training loss of the second stage is formulated as the weighted combination of the two losses:

where λ decides the relative importance of the two losses.

3 Experiments

3.1 Experimental Setup

Datasets. We evaluate the performance of our approach on two widely-used benchmark datasets: BP4D [Zhang et al., 2014] and DISFA [Mavadati et al., 2013]. BP4D recorded 328 videos (about 140,000 facial frames) from 41 young adults (23 females and 18 males) who were asked to respond to 8 emotion elicitation tasks. DISFA recorded 130, 815 frames from 27 subjects (12 females and 15 males) who were watching Youtube videos. Each frame in BP4D and DISFA is annotated with occurrence labels of multiple AUs.

Implementation Details. For both datasets, we use MTCNN [Yin and Liu, 2017] to perform face detection and alignment for each frame and crop it to 224 × 224 as the input for backbones. We then follow the same protocol as previous studies [Zhao et al., 2016; Li et al., 2018; Song et al., 2021c] to conduct subject-independent three folds cross-validation for each dataset, and report the average results over 3 folds. During the training, we employ an AdamW optimizer with β1 = 0.9, β2 = 0.999 and weight decay of 5e-4 . The number K for choosing nearest neighbors in the FGG is set to 3 and 4 for BP4D and DISFA, respectively. For the hyperparameter λ in Eq. 8, we set it to 0.05 and 0.01 for models based on ResNet and Swin Transformer, respectively. We totally train the proposed model for 40 epochs, including 20 epochs for the first stage (the initial learning rate of 1e-4 ) and 20 epochs for the second stage (the initial learning rate of 1e-6 ), with a batch size of 64. The cosine decay learning rate scheduler is also used. Both backbones are pre-trained on ImageNet [Deng et al., 2009].

Evaluation Metric. We follow previous AU occurrence recognition studies [Shao et al., 2021a; Churamani et al., 2021; Li et al., 2019b; Song et al., 2021c] using a common metric: frame-based F1 score, to evaluate the performance of our approach, which is denoted as F1 = 2 P·R P+R . It takes the recognition precision P and recall rate R into consideration.

3.2 Results and Discussion

Comparison to State-of-the-art Methods. This section compares our best systems of two backbones with several state-of-the-art methods on both datasets. Table 1 reports the occurrence recognition results of 12 AUs on BP4D. We additionally provide the AUC results in Appendix A. It can be observed that the proposed AU relationship modelling approach allows both backbones (ResNet-50 and Swin TransformerBase (Swin-B)) to achieve superior overall F1 scores than all other listed approaches, with 0.5% and 1.3% average improvements over the state-of-the-art [Jacob and Stenger, 2021]. Specifically, our approach allows both backbones to achieve the top three performances for 9 out of 12 AUs' recognition (e.g., AU 4, AU 6, AU 7, AU 10, AU 12, AU 14, AU 15, AU 17, and AU 23) among all listed approaches. Similar results were also achieved on DISFA. According to Table 2, our approach helps both backbones to achieve the state-of-the-art average F1 scores over 8 AUs, which outperform the current state-of-the-art with 1.6% and 0.9% improvements, respectively. For fair comparisons, we only compare our approach with static face-based methods that did not remove any frame from the datasets.

According to both tables, our ResNet-50-based system also clearly outperforms other graph-based AU recognition approaches which also use CNNs (ResNet (UGN-B, HMP-PS) or VGG (SRERL)) as backbones. Since SRERL only uses a pre-computed adjacent matrix to describe the relationship between AUs for all faces, our system shows a large advantage over it, with 1.8% and 7.2% F1 score improvements for the average results on BP4D and DISFA, respectively. Although UGN-B and HMP-PS assigned each facial display a unique adjacent matrix and achieved better performance than SRERL, they still use a single value to describe the relationship between each pair of AUs, without considering multiple relationship cues. Thus, our deep-learned task-specific multi-dimensional edge features lead our system to achieve more than 1.3% and 2.1% average F1 score improvements over UGN-B and HMP-PS on both datasets.

Ablation Studies. Table 3 evaluates the influence of each component of our pipeline on the average AU recognition results. It can be observed that simply using the AFG block to specifically learn a representation for each AU enhanced the performance for both backbones, indicating that the relationship between each AU's activation and the full face representation is unique. In particular, when a facial AU is activated, its movement usually affects other facial regions (i.e., the activation of other AUs) while inactivated AUs would not have such an effect. As visualized in Figure 4, our FGG simulates this phenomenon by connecting activated AUs to all other AUs (including activated and inactivated AUs). Building on the backbone-AFG system, we also found that individually adding the FGG block or MEFL module further increased the recognition performance for both backbones. These results suggest that (i) the FGG block allows the AFG block to encode additional AU recognition-related cues into node features, i.e., we hypothesize that the FGG can help the AFG to learn AUs' relationship cues for their recognition; and (ii) the multi-dimensional edge features learned by the MEFL module provide more task-specific AU relationship cues to improve the recognition performance, which further validates our hypothesis that a single value is not enough to carry all useful relationship cues between a pair of AUs.

In short, the proposed approach can provide valuable relationship cues for AU recognition during both node and edge feature learning. More importantly, jointly using FGG and MEFL with our weighted asymmetric loss largely boosted both backbones' recognition capabilities, i.e., 5.6% and 2.9% F1 score improvements over the original backbones, as well as 1.7% and 0.9% improvements over the backbone-AFG systems. Besides the proposed relationship modelling approaches, we show that the two loss functions also positively improved the recognition performance. The weighted asymmetric loss clearly enhanced the performance over the widely-used weighted binary cross-entropy loss, illustrating its superiority in alleviating data imbalance issue. Meanwhile, the proposed AU co-occurrence supervision also slightly enhanced recognition results for both backbones.

4 Conclusion

This paper proposes to deep learn a graph that explicitly represents relationship cues between each pair of AUs for each facial display. These relationship cues are encoded in both node features and multi-dimensional edge features of the graph. The results demonstrate that the proposed node and edge feature learning methods extracted reliable task-specific relationship cues for AU recognition, i.e., both CNN and transformer-based backbones have been largely enhanced, and achieved state-of-the-art results on two widely used datasets. Since our graph-based relationship modelling approach can be easily incorporated with standard CNN/transformer backbones, it can be directly applied to enhance the performance of multi-label tasks or tasks whose data contains multiple objects, by explicitly exploring the task-specific relationship cues among labels or objects.

A Additional Experimental Results

Acknowledgements

Cheng Luo, Weicheng Xie and Linlin Shen are supported by the National Natural Science Foundation of China under grants no. 61602315, 91959108, the Science and Technology Project of Guangdong Province under grant no. 2020A1515010707, the Science and Technology Innovation Commission of Shenzhen under grant no. JCYJ20190808165203670. Siyang Song is supported by the European Union's Horizon 2020 Research and Innovation programme under grant agreement No. 826232. Hatice Gunes is supported by the EPSRC under grant ref. EP/R030782/1.