Dual-Cross Central Difference Network for Face Anti-Spoofing

1 Introduction

Face recognition technology has widely used in many interactive intelligent systems due to their convenience and remarkable accuracy. However, face recognition systems are still vulnerable to presentation attacks (PAs) ranging from print, replay and 3D-mask attacks. Therefore, both the academia and industry have recognized the critical role of face antispoofing (FAS) for securing the face recognition system.

∗Corresponding author. This work was supported by the Academy of Finland for project MiGA (grant 316765), ICT 2023 project (grant 328115), Infotech Oulu, project 6+E (grant 323287) funded by Academy of Finland, and project PhInGAIN (grant 200414) funded by The Finnish Work Environmental Fund. The authors wish to acknowledge CSC-IT Center for Science, Finland, for computational resources.

[de Freitas Pereira et al., 2012; Komulainen et al., 2013; Patel et al., 2016] and deep learning-based [Yu et al., 2020d; Yu et al., 2020a; Liu et al., 2018; Jourabloo et al., 2018; Yang et al., 2019; Yu et al., 2020b] methods have shown effectiveness for presentation attack detection (PAD). On one hand, some classical local descriptors (e.g., local binary pattern (LBP) [Boulkenafet et al., 2015] and histogram of gradient (HOG) [Komulainen et al., 2013]) are robust for describing the detailed invariant information (e.g., color texture, moir´e pattern and noise artifacts) from spoofing faces. However, the shallow and coarse feature extraction procedure limits the discriminative capacity of these local descriptors.

On the other hand, convolutional neural networks (CNNs) focus on representing deeper semantic features to distinguish the bonafide and PAs, which are weak in capturing finegrained intrinsic patterns (e.g., lattice artifacts) between the live and spoof faces, and easily influenced under diverse scenarios. Although central difference convolution (CDC) [Yu et al., 2020d] introduced the central-oriented local gradient features to enhance models' generalization and discrimination capacity, it still suffers from two disadvantages. In CDC, central gradients from all neighbors are calculated, which is 1) inefficient in both inference and back-propagation stages; and 2) redundant and sub-optimized due to the discrepancy among diverse gradient directions. Thus, to study the impacts and relations among central gradients is not a trivial work.

One key challenge in the FAS task is how to learn representation with limited data as existing FAS datasets (e.g., OULUNPU [Boulkenafet et al., 2017] and SiW-M [Liu et al., 2019]) do not have large amount of training data due to the high collection cost for both spoofing generation and video recording. Although generic augmentation manners (e.g., horizontal flip, color jitter and Cutout [DeVries and Taylor, 2017]) are able to expand the scale and diversity of live/spoof samples, it still contributes not much performance improvement. Thus, it is worth rethinking the augmentation for FAS and design taskdedicated augmentation paradigm.

Motivated by the discussions above, we propose a novel convolution operator family called Cross Central Difference Convolution (C-CDC), which decouples the central gradient features into cross-directional combination (horizontal/vertical or diagonal) thus more efficient and concentrated for message aggregation. Furthermore, in order to mimic more general attacks (e.g., partial print and mask attacks) and learn to distinguish spoofing in both global and patch level, Patch Exchange (PE) augmentation is proposed for mixed sample as well as corresponding dense label generation. To sum up, our contributions include:

• We design a sparse convolution family called Cross Central Difference Convolution (C-CDC), which decouples the vanilla CDC into two cross (i.e., horizontal/vertical and diagonal) directions, respectively. Compared with CDC, our proposed C-CDC could achieve better performance for FAS with only five ninth parameters and less computational cost.
• We propose a Dual-Cross Central Difference Network (DC-CDN), consisting of two-stream backbones with horizontal/vertical and diagonal C-CDC, respectively. Moreover, we also introduce Cross Feature Interaction Modules (CFIM) between two streams of DC-CDN for mutual neighbor relation mining and local detailed representation enhancement.
• We propose the first FAS-dedicated data augmentation method, Patch Exchanges (PE), to synthesize mixed samples with diverse attacks and domains, which is able to plug and play in not only DC-CDN but also existing FAS methods for performance improvement.
• Our proposed method achieves state-of-the-art performance on four benchmark datasets with intra-dataset, cross-dataset, and cross-type testing protocols.

2 RelatedWork

Face Anti-Spoofing. Traditional face anti-spoofing methods usually extract handcrafted features from the facial images to capture the spoofing patterns. Some classical local descriptors such as LBP [Boulkenafet et al., 2015] and HOG [Komulainen et al., 2013] are utilized for handcrafted features. More recently, a few deep learning based methods are proposed for face anti-spoofing. On the one hand, FAS can be naturally treated as a binary classification task, thus binary cross entropy loss is used for model supervision. On the other hand, according to the physical discrepancy between live and spoof faces, dense pixel-wise supervisions [Yu et al., 2021] such as pseudo depth map [Liu et al., 2018; Yu et al., 2020d; Wang et al., 2020], reflection map [Yu et al., 2020a], texture map [Zhang et al., 2020] and binary map [George and Marcel, 2019] are designed for fine-grained learning. In this work, we supervise the deep networks with pseudo depth map due to its effectiveness.

Due to the high collection cost for spoof attacks, there are limited data scale and diversity in public datasets. Supervised with small-scale predefined scenarios and PAs, most existing FAS methods are easy to overfit and vulnerable to domain shift and unseen attacks. In order to detect unseen attacks successfully, deep tree network [Liu et al., 2019] and adaptive inner-update meta learning [Qin et al., 2020] are developed for zero-shot FAS. However, it is still urgent to provide larger-scale and richer live/spoof data for deep models training. Here we consider novel data augmentation for FAS to tackle this challenge.

Convolution Operators. The convolution operator is commonly used for local feature representation in modern deep learning framework. Recently, a few extensions to the vanilla convolution operator have been proposed. In one direction, pre-defined or learnable local relation is embedded in the convolution operator. Representative works include Local Binary Convolution [Juefei-Xu et al., 2017] and Gabor Convolution [Luan et al., 2018], which is proposed for local invariance preservation and enhancing the resistance to spatial changes, respectively. Besides, self-attention layer [Parmar et al., 2019], self-attention block [Zhao et al., 2020] and local relation layer [Hu et al., 2019] are designed for mining the local relationship flexibly. Another direction is to modify the spatial scope for aggregation. Two related works are dialated convolution [Yu and Koltun, 2015] and deformable convolution [Dai et al., 2017]. However, these convolution operators may not be suitable for FAS task because of the limited representation capacity for invariant fine-grained features. In contrast, CDC [Yu et al., 2020d; Yu et al., 2020c; Yu et al., 2020b] is proposed for invariant and detailed features extraction, which is suitable for the FAS task. In this paper, we devote to improving the vanilla CDC in terms of both performance and efficiency.

3 Methodology

In this section we first briefly review the CDC [Yu et al., 2020d], and then introduce the novel Cross Central Difference Convolution (C-CDC) family in Sec. 3.2. Based on the C-CDC operators, we propose Dual-Cross Central Difference Networks in Sec. 3.3. Finally we present the novel data augmentation strategy in Sec. 3.4.

3.1 Preliminary

As the basic operator in deep networks, the vanilla 2D convolution consists of two main steps: 1) sampling local neighbor region R over the input feature map x; and then 2) aggregating the sampled values via learnable weights w. As a result, the output feature map y can be formulated as

where p0 denotes the current location on both input and output feature maps while pn enumerates the locations in R. For instance, local receptive field region for convolution operator with 3×3 kernel and dilation 1 is R = {(-1,-1), (-1, 0), · · · , (0, 1), (1, 1)}.

Different from the vanilla convolution, the CDC introduces central gradient features to enhance the representation and generalization capacity, which can be formulated as

As both intensity-level semantic information and gradientlevel detailed clues are crucial for robust FAS, the generalized CDC operator can be represented by combination of vanilla convoluiton and CDC

where hyperparameter θ ∈ [0, 1] trade-offs the contribution between intensity-level and gradient-level information. The higher value of θ means the more importance of central gradient features. Please note that w(pn) is shared between vanilla convolution and CDC, thus no extra parameters are added. Henceforth, the generalized CDC will be referred as CDC directly for clear statement.

3.2 Cross Central Difference Convolution

As can be seen from Eq. (3) that CDC aggregates both the vanilla and central gradient features from entire local neighbor region R, which might be redundant and hard to be optimized. We assume that exploiting sparse center-oriented difference features would alleviate local competition and overfitting issues. Therefore, we propose the sparse Cross Central Difference Convolution (C-CDC) family, intending to learn more concentrated and intrinsic features for FAS.

Compared with CDC operating on R, C-CDC prefers to sample a sparser local region S, which can be formulated as

To be specific, we decouple R into two cross neighbor regions, including 1) horizontal & vertical (HV) cross neighbor regions SHV = {(-1, 0), (0, -1), (0, 0), (0, 1), (1, 0)}; and 2) diagonal (DG) cross neighbor regions SDG = {(-1, -1), (-1, 1), (0, 0), (1, -1), (1, 1)}. In this way, horizontal & vertical C-CDC and diagonal C-CDC can be represented when S = SHV and S = SDG, respectively. Figure 1 illustrates the workflow of the C-CDC(HV) and C-CDC(DG). Similarly, the generalized C-CDC can be easily formulated when replacing R with S in Eq. (3). We will use this generalized C-CDC henceforth.

In terms of designs of the sparse local region S, there are also some other solutions with different neighbor locations or fewer neighbors. The reason that we consider the cross (i.e., HV and DG) fashions for S derives from their symmetry, which is beneficial for model convergence and robust feature representation. The studies about the sparsity and neighbor locations are shown in Section 4.3.

3.3 Dual-Cross Central Difference Network

Pseudo depth-based supervision takes advantage of the discrimination between live and spoof faces based on 3D shape, which is able to provide pixel-wise detailed clues to enforce FAS model to capture intrinsic features. Following the similar depth-supervised backbone as "DepthNet" [Liu et al., 2018] and "CDCN" [Yu et al., 2020d], we replace all the 3 × 3 convolution operators with our proposed C-CDC to form the Cross Central Difference Network (C-CDN). Given a single RGB face image with size 3 × 256 × 256, multi-level (lowlevel, mid-level and high-level) fused features are extracted for predicting the grayscale facial depth with size 32×32. The details of C-CDN are shown in Table 1. It can be seen that with the similar architecture (e.g., network depth and width), C-CDN has only five ninth parameters and two hundredth computational cost compared with DepthNet and CDCN due to the sparse local sampling mechanism in C-CDC. We use θ = 0.8 as the default setting, and the corresponding study about θ will be discussed in Section 4.3.

Although the C-CDC decouples and learns the local gradient features with particular views (HV and DG), it still suffers from information loss compared with CDC operating on the full local neighbors. In order to fully exploit the local features and interact between HV and DG views, a DualCross Central Difference Network (DC-CDN) is proposed. As shown in Figure 2, two independent (unshared) networks respectively assembled with C-CDC(HV) and C-CDC(DG) are used. Then the extracted dual-stream features from different views are fused for final depth prediction. In this way, the full neighbor aggregation step can be disentangled into two sub-steps: 1) sparse neighbor aggregation for individual stream; and 2) dual-stream fusion.

Cross Feature InteractionModule. With only simple late fusion, the performance improvement might be limited due to the lack of message passing from the preceding (i.e., lowlevel, mid-level, and high-level) stages. In order to effectively mine the relations across dual streams and enhance local detailed representation capacity, we propose the Cross Feature Interaction Module (CFIM) to fuse dual-stream multi-level features adaptively. To be specific, given the two-stream features FHV and FDG, the CFIM enhanced features F ′HV and F ′DG can be formulated as

where ϕ() denotes the Sigmoid function for [0,1] range mapping. α and β are the attention weights for FHV and FDG, respectively. In our default setting, both α and β are initialized to 0 and learnable, which could be adaptively adjusted during the training iterations. As illustrated in Fig. 2, here we respectively plug three CFIMs (with learnable αlow, αmid, αhigh, βlow, βmid, βhigh) in the output features from lowmid-high levels before multi-level concatenation.

Algorithm 1 Patch Exchange Augmentation

Input: Face images I with batchsize N, pseudo depth map labels D, augmented ratio γ ∈ [0, 1], step number ρ

1 : for each Ii and Di, i = 1, ..., ⌊γ ∗N⌋ do

• 2 : for each step ρ do
• 3 : Randomly select a patch region P within Ii
• 4 : Randomly select a batch index j, j ≤ N
• 5 : Exchange the image patch Ii(P) = Ij(P) and label patch Di(P ) = Dj (P )

6 : end

7: end

8: return augmented I and D

3.4 Patch Exchange Augmentation

Due to the high collection cost for spoof attacks, there are limited data scale and diversity in public FAS datasets. In this paper, we also propose a FAS-dedicated data augmentation method, named Patch Exchanges (PE), to synthesize mixed samples with diverse attacks and domains. There are three advantages for PE augmentation: 1) face patches from different domains (e.g., recorded scenario, sensor, and subject) are introduced for enriching data distribution; 2) random live and PA patches are exchanged to mimic arbitrary partial attacks; and 3) the exchanged patches with corresponding dense labels enforce the model to learn more detailed and intrinsic features for spoofing detection. The complete algorithm of PE is summarized in Algorithm 1. As a tradeoff, we use empirical settings γ = 0.5 and ρ = 2 for experiments.

Note that as the face images I for PE are coarsely aligned, the exchanged patches would have similar semantic content (e.g., cheek, nose and mouth) but with diverse live/spoof clues. Thus, the augmented live/spoof faces are still realistic and even more challenging to be distinguished. Some typical samples with PE augmentation are visualized in Figure 3.

4 Experiments

4.1 Datasets andMetrics

Databases. Four databases OULU-NPU [Boulkenafet et al., 2017], CASIA-MFSD [Zhang et al., 2012], ReplayAttack [Chingovska et al., 2012] and SiW-M [Liu et al., 2019] are used in our experiments. OULU-NPU is a highresolution database, containing four protocols to evaluate the generalization (e.g., unseen illumination and attack medium) of models respectively, which is used for intra testing. CASIA-MFSD and Replay-Attack are small-scale databases with low-resolution videos, which are used for cross testing. SiW-M is designed for cross-type testing for unseen attacks as there are rich (13) attack types inside.

Performance Metrics. In OULU-NPU dataset, we follow the original protocols and metrics, i.e., Attack Presentation Classification Error Rate (APCER), Bona Fide Presentation Classification Error Rate (BPCER), and ACER for a fair comparison. Half Total Error Rate (HTER) is adopted in the cross testing between CASIA-MFSD and Replay-Attack. For the cross-type test on SiW-M, ACER and Equal Error Rate (EER) are employed for evaluation.

4.2 Implementation Details

Depth Generation. Dense face alignment [Guo et al., 2020] is adopted for pseudo depth generation. To clearly distinguish live and spoof faces, at the training stage, we follow [Liu et al., 2018] to normalize the live depth maps in a range of [0, 1], while leaving spoof depth maps to all zeros.

Training andTesting Setting. Our proposedmethod is implemented with Pytorch. In the training stage, models are trained with batch size 8 and Adam optimizer on a single V100 GPU. Data augmentations including horizontal flip, color jitter and Cutout are used as baseline. The initial learning rate (lr) and weight decay are 1e-4 and 5e-5, respectively. We train models with maximum 800 epochs while lr halves in the 500th epoch. Similar to [Yu et al., 2020d], all the models are supervised by mean square error (MSE) and contrastive depth loss (CDL). In the testing stage, we calculate the mean value of the predicted depth map as the final score.

4.3 Ablation Study

In this subsection, all ablation studies are conducted on the Protocol-1 of OULU-NPU dataset.

Impact of θ in C-CDC. As discussed in Section 3.2, θ controls the contribution of the gradient-based features, i.e., the higher θ, the more local detailed information included. As illustrated in Fig. 4(a), when θ ⩾ 0.3, C-CDC(HV) and C-CDC(DG) always achieve better performance than their vanilla counterpart (i.e., θ = 0), indicating the effectivenss of local gradient features for FAS task. As the best performance (ACER=0.7% and 0.9% for C-CDC(HV) and C-CDC(DG), respectively) are obtained when θ = 0.8, we use this setting for the following experiments.

C-CDC vs. CDC. Here we evaluate the impacts of the neighbor sparsity for both vanilla and gradient-based convolutions. As shown in Fig. 4(b), "VanillaConv(HV)" and "VanillaConv(DG)" performs more poorly than "VanillaConv", which might be caused by the limited semantic representation capacity from the sparse neighbor sampling. In contrast, it is interesting to see that their central differencebased counterparts perform inversely. Compared with CDC, C-CDC(HV) and C-CDC(DG) have only five ninth parameters, and even achieve better performance (-0.3% and -0.1%

ACER, respectively), indicating the efficiency of cross gradient features for FAS.

Effectiveness of DC-CDN and CFIM. The upper part of Table 2 shows the ablation results of DC-CDN w/ and w/o CFIM. It can be seen from the third and fourth rows that DC-CDN w/o CFIM achieves worse when using simple final depth map averaging or late fusion via concatenation. In contrast, assembled with CFIM, DC-CDN obtains 0.2% and 0.4% ACER decrease compared with one-stream CCDN(HV) and C-CDN(DG), respectively. It demonstrates the importance of adaptive mutual feature interaction during the training stage.

Effectiveness of PE augmentation. It can be seen from the lower part of Table 2 that PE augmentation improves the performance of both single-stream (C-CDN) and dualstream (DC-CDN) models on Protocol 1 (with slight domain shifts) of OULU-NPU. It is worth noting that, benefited from the augmented data with rich and diverse domain and attack clues, DC-CDN could obtain remarkable performance gains on more challenging senarios, including cross-type (-1.2% EER, see Table 4) and cross-dataset (respective -2.5% and -2.2%HTER, see Table 5) testings.

4.4 Comparison with State of the Arts

Intra Testing on OULU-NPU. As shown in Table 3, our proposed DC-CDN ranks first on the first three protocols (0.4%, 1.3% and 1.9% ACER, respectively), which indicates the proposed method performs well at the generalization of the external environment, attack mediums and input camera variation. It is clear that the proposed DC-CDN consistently outperforms CDCN [Yu et al., 2020d] on all protocols with -0.6%, -0.2%, -0.4%, and -2.9%, respectively, indicating the superiority of DC-CDN. In the most challenging Protocol 4, DC-CDN also achieves comparable performance with stateof-the-art SpoofTrace [Liu et al., 2020] but more robustness with smaller ACER standard deviation among six kinds of unseen scenarios.

Cross-type Testing on SiW-M. Following the leave-onetype-out (total 13 attack types) protocol on SiW-M, we compare the proposed methods with several recent FAS methods to validate the generalization capacity of unseen attacks. As shown in Table 4, our method achieves the best EER performance and can perform more robustly in several challenging attacks including 'Silicone Mask', 'Transparent Mask', 'Mannequin', 'Impersonation', 'Funny Eye' and 'Paper Glasses'. Note that our DC-CDN could achieve comparable performance with the SOTA method BCN [Yu et al., 2020a], which is supervised by three kinds of pixel-wise labels. Moreover, it is reasonable to see from the last two rows of Table 4 that PE augmentation helps to improve generalization ability of unknown Partial Attacks obviously.

Cross-dataset Testing. Here cross-dataset testing is conducted to further testify the generalization ability of our models under unseen scenarios. There are two cross-dataset testing protocols. One is that training on the CASIA-MFSD and testing on Replay-Attack, which is named as protocol CR; the second one is exchanging the training dataset and the testing dataset, named protocol RC. As shown in Table 5, our proposed DC-CDN achieves 6.0% HTER on protocol CR, outperforming the prior state-of-the-art by a convincing margin of 9.5%. For protocol RC, we also slightly outperform stateof-the-art frame-level methods (e.g., CDCN and BCN).

5 Conclusions

In this paper, we propose a sparse operator family called Cross Central Difference Convolution (C-CDC) for face antispoofing task. Moreover, we design a Dual-Cross Central Difference Network with Cross Feature Interaction Modules for dual-stream feature enhancement. Besides, a novel Patch Exchange augmentation strategy is proposed for enriching the training data. Extensive experiments are performed to verify the effectiveness of the proposed methods.