Live Face Verification with Multiple Instantialized Local Homographic Parameterization

1 Introduction

Live face verification has been a very important building block for modern face recognition applications such as online banking, online shopping etc. The purpose of live face verification is to recognize whether the face under examination is being captured from a real person or some types of , e.g., showing the recorded face image/video on a pad to the camera. Previous methods [Pan et al., 2007] mainly adopt an strategy to distinguish a real face from a fake one. Namely, the recognition systems ask the user to present different facial expressions such as smile, shaking head, close/open eye and then perform emotion recognition to test whether the predicted emotion matches the given order. However, such a strategy has inherent drawbacks. First, completing the entire verification process usually requires several minutes, which is NOT user friendly. Second, such a system could be very easily attacked by recording down all possible emotions of the user. To this end, recent works [Tan et al., 2010;

∗ corresponding author: Bingbing Ni.

†Shanghai Institute for Advanced Communication and Data Science, Shanghai Key Laboratory of Digital Media Processing and Transmission, Shanghai Jiao Tong University, Shanghai 200240, China.

t + 1

Figure 1: Visualization of our motivations. Each video is a bag containing 9 patches. Each patch is an instance, from which a motion map is extracted. The homography motion maps are labeled with yellow stars. With the motion maps, we can infer that the video in the first row is a real face video and the video in the second row is an attack face video.

Yang et al., 2013; Kollreider et al., 2005] have started exploring a solution.

Namely, users are allowed to perform any facial movements in a natural way, and the algorithm extract static motion features based on a single frame or several consecutive frames, to directly perform verification. No emotion actions are required.

Although more user friendly, it is very challenging for this method to achieve very low miss detection rate as well as false alarm rate, due to the following reasons. Most of these methods extract various holistic image features (e.g., such as Local Binary Patterns [M¨a¨att¨a et al., 2011], 3D face shape analysis [Lagorio et al., 2013], Fourier spectra [Li et al., 2004]) or motion features between frames (e.g., such as optical flow [Bao et al., 2009] motion magnification [Bharadwaj et al., 2013] etc.) and then perform ensemble learning for improving the verification performance on a certain dataset. This approach is harmful for at least two reasons. First, as no physical geometric prior knowledge is used for exploring informative static/motion features, the learned features might not necessary correspond to the underlying between a real face and a fake one, i.e., the learned features might overfit. Second, these methods usually treat the entire image as a whole, however, to recognize faces discriminatively, local subtle features are more helpful. For example, [Sun et al., 2014] extract features from multiple patches to do face identification. Using global feature representations might degrade the sensitivity of these algorithms. This work proposes a patch-wise motion parameterization based verification network to explicitly address these issues. This network is motivated by two observations. First, we note that although for the entire face, there exist NO explicit (specific) motion patterns when the user is performing certain emotions, there DO exist some local motion patterns when the faces are viewed from a planer objects (e.g., pad). In a short time-span, local facial patches could be approximately considered as a rigid planer object, therefore local homography holds. This motivates us to construct a network which extracts local motion maps (shown in Figure 1) from local facial patches (in our cases, a STN [Jaderberg et al., 2015] network component), therefore to distinguish a real face from a fake one by attending to local motion patterns. Furthermore, inspired by the fact that, we embed our network into a multiple instance learning framework. In this sense, local subtle facial movements is more reasonable for our model, and therefore further enhance the recall rate of the proposed technique. Extensive experimental results on several face benchmarks well demonstrate the superior performance of our method.

The rest of this paper is organized as follows. We first review the previous work in three fields: the live face verification, the homography transformation, and the multiple instance learning. Then, we design a method to do the face verification task. Finally, we perform extensive experiments on several benchmarks with in-depth analysis.

2 RelatedWork

In live face verification , many motion features based methods [Li et al., 2004; Bharadwaj et al., 2013; Bao et al., 2009] try to detect the live face clues. However, these live clues could also be easily detected from a replaying planar screen. On the contrary, we focus on finding "fake" clues by detecting the homography holding regions. If we find any "fake" face clues in a video, we think it is an attack video.

Two consecutive frames are related by homography if the captured object is a rigid body [Szeliski, 2004]. For a whole frame, global homography doesn't hold when there contains moving objects while the background stays static. However, we can respectively explore homography for each local patch. Early homography detecting methods, such as [Hinterstoisser et al., 2008], usually first detect the matching pixels between two consecutive frames. In our case, the matching process will lead to computing cost. To this end, we utilize the Spatial Transform Network (STN) [Jaderberg et al., 2015]. It meets our conditions for the following reasons: 1) to estimate the homography, the first thing is to settle down the corresponding homography matrix, which is determined by six parameters. The six parameters allow rotation, translation and scaling. The STN which trains six warp parameters, is usually applied to do affine transformation; 2) the STN embedded to Convolutional Neural Network (CNN) don't need the process of finding the matching points between two frames.

multiple instance learning (MIL) has improved the performances of many tasks in computer vision including segmentation [Yang et al., 2017], tracking [Babenko et al., 2009] and posing detection [Babenko et al., 2008], since [Dietterich et al., 1997]. MIL collects all instances into a bag, and one instance is not singleton. A bag is positive if there exists at least one positive instance; otherwise it is negative. In our work, we collect several patches from a whole frame, if we find any homography patches, we can regard the video as an attacking video and this naturally meets the MIL conditions.

3 Method

3.1 Motivation and Overview

As introduced above, we are motivated by the following two observations:

• O1: Local facial patches from the recorded video present significant motion pattern, e.g., local homographic property between adjacent frames, while real ones do not possess this nature.
• O2: For a fake face, there exist multiple patches which are easily recognized as "fake", while all local patches of a real one must be recognized as "real".

Following these two motivations shown in Figure 1, our proposed framework is detailed as follows. The input is two adjacent frames containing face instances. After face detection for each frame, each face is resized to the fixed size and then divided equally into 9 local patches. So we obtain nine pairwise local facial patches from two adjacent frames. All pairwise patches are input into a Local Motion Discovery Module which explores the underlying motion characteristic and outputs some local motion parameterization maps. Then, each of these local motion parameterization maps concatenated with the corresponding pair-wise local facial patch is input into a classification sub-network to evaluate whether the pair-wise patch is from a real video or not. To take full advantage of O2, we consider each face as a bag of local patches and propose a multiple instance based novel loss function to improve the recall rate of attack video. Our proposed network is illustrated in Figure 2.

3.2 Prerequisite

Live face verification can be defined as a two classification problem, that is, given a video containing a face, to determine whether the video is a real video or an attack video. Let {(V tr i , yi)}Ni=1 denotes the training set, where V tr i represents a training face video, yi is the related ground truth, and N is the number of videos in the training set. yi ∈ {0, 1}, 0 represents real video and 1 represents attack video. First, the videos are disassembled into frames, i.e., Vi = {Fi,j}Mi j=1 , where Fi,j denotes the j-th frame in video Vi, Mi is the number of frames in Vi. Then, we get the bounding boxes of face regions in the whole frames by applying the Dlib's face detection algorithm. We further crop the face regions and resize them to 192 × 192. In order to get local subtle features of the faces, we divide the face into 9 average patches, Fi,j = {p1i,j , . . . , pki,j , . . . , p9i,j } shown in Figure 1.

3.3 Local Motion DiscoveryModule

We use the planar homography to parameterize local motion patterns. A planar homography is a transformation from one rigid plane to another, which is very useful to estimate the image transformation. Images of points on a plane in one view are related to corresponding image points in another view by a planar homography.

A planar homography can be represented as a 3×3 matrix. The coordinate transformation is

where (x1, y1) and (x2, y2) are the corresponding coordinates for a rigid planar object in two different views.

In the live face verification application, the frames can be regarded as different views of a face, due to the relative motion between the camera and the captured face. For a real face video, because the captured face is not a rigid plane, the corresponding coordinates in different frames do not satisfy the transformation Eq.(1), i.e., the frames are not related by homography. While for a re-captured face video, the re-playing screen or the attacking photo is a rigid plane, so all the frames are related by homography. But when the user is performing certain emotions, there exists no explicit global homography patterns. An example is shown in Figure 3. However, in a short time span, some facial patches could be approximately considered as a rigid planar object, there do exist some local homography patterns for these patches. Thus the local homography character is a useful feature in the live face verification task.

In order to extract local motion parametrization features from local facial patches, we develop a local motion discovery module similar to the Spatial Transform Network (STN) [Jaderberg et al., 2015], which is shown in Figure 2. The local motion discovery module consists of a corresponding points matching sub-network, a transformation sub-network and a classification sub-network. Different from the original STN which takes one image as input, we input two patches pki,j and pki,j+1 , which are the k-th patches from consecutive j-th and j+1-th frames, to this module.

• The matching sub-network is a regular CNN. Its input is the two patches pki,j and pki,j+1 and it outputs a homography matrix.
• The transformation sub-network has no trainable parameters. It only applies the homography matrix to patch pki,j , and obtains a prediction patch ̂ pki,j+1 . The output of this sub-network is the difference between the pre-

where Xk i,j represents the local motion parametrization features of the k-th local patch.

• The classification sub-network consists of a series of convolution layers, pooling layers and ends with a fully connected layer following a softmax layer. The input of the classification sub-network is a concatenation of Xi,j , pki,j and pki,j+1 . It outputs two probabilities indicating whether the two patches are from a real face video or an attack video. The classification sub-network is supervised by the ground truth yi for video V tr i . The gradient is back propagated to both the weights of the classificat0ion sub-network and the matching sub-network.

It is worth noting that our matching sub-network is never explicitly learned from the two local patches. Instead, it learns automatically the homography transformation parameters that directly enhances the final classification accuracy.

3.4 Multi-Patch ExaminationModule

In the local motion discovery module, each face video is divided into 9 pair-wise local facial patches. We can regard each video

as a bag, where (pki,j , pki,j+1 ) represents an instance, and pki,j is the k-th patch of j-th frame in video Vi. Note that all instances(local patches) in a real face bag must be recognized as "real", therefore all the instances labels are available. But for a fake face, not all instances are recognized as "fake", because the homography motion patterns don't exist in some patches. The labels of instances in fake face bags are unknown for we Algorithm 1 Learning algorithm for multi-patch examination module

Input: Training bags Btr = {(V tr i , yi)}Ni=1 , where V tr i represents a training video and yi ∈ {0, 1}

Output: Multiple instance model

• 1: Initialize instance labels {y1 i , · · · , yk i , · · · , y9 i } with the value of yi
• 2: Train local motion discovery module with cross-entropy loss shown by Eq. 3 for one epoch and get a pre-trained model;
• 3: repeat
• 4: Input one batch data in the pre-trained model and output the probabilities {Pr1 i , · · · , Prk i , · · · , Pr9 i } for every instance of V tr i ;
• 5: for each sample in the batch do
• 6: if yi = 1 then 7: yk i ← 1(Prk i > 0.5) for every instance label in {y1 i , · · · , yk i , · · · , y9 i };
• 8: else
• 9: {y1 i , · · · , yk i , · · · , y9 10: end if
• 11: end for
• 12: Use {y1 i , · · · , yk i , · · · , y9 i } as ground truth and train local motion discovery module
• 13: if iterating over an epoch then
• 14: Update the pre-trained model with current model.
• 15: end if
• 16: until max iteration is reached.
• 17: Output the final multiple instance model.

don't know which patches are "fake" in the bags. To find out the specific "fake" patches, we embed our work into a multiple instances framework.

Multiple Instances Learning (MIL) deals with the task in which the bag labels are available, whereas instance labels are unknown. A bag is positive if it contains at least one positive instance, and negative otherwise. The positive instances are also called the key instances. We propose a multiple instance based novel loss function which is embedded into the local motion discovery module to obtain an end-to-end multiple instance learning framework.

Algorithm 1 shows the details of learning process. The learner is given a training data set with N bags Btr = {(V tr 1 , y1) , . . . , (V tr i , yi) , . . . , (V tr N , yN)}, where V tr i = {(p1i,j , p1i,j+1 ) , . . . , (pki,j , pki,j+1 ) , . . . , (p9i,j , p9i,j+1 )} is a bag with (pki,j , pki,j+1 ) representing an instance and pki,j is the k-th patch of j-th frame in video V tr i . yi ∈ {0, 1} is the label of V tr i , where 1 represents the bag is a "fake" face video and 0 represents the bag is a "real" face video. If there exists any positive instance in bag V tr i , then V tr i is a positive bag ("fake" face video) with yi = 1; otherwise it is a negative bag ("real" face video) with yi = 0. Our goal is to find the positive instances, that is, the homography motion pattern patches. For the classification sub-network in the local motion discovery module, the cross-entropy loss for one batch data can be expressed as

(3)

where B represents batch size; P is the number of patches, which is equal to 9 in our case. yk i is the instance label. Prk i is the probability of V tr i as a "fake" video predicted by the classification sub-network. Our multiple instance based loss function adjusts the labels yk i of the instances.

In the first epoch, we initialize the instance labels {y1 i , · · · , yk i , · · · , y9 i } with the value of yi and update parameters of the local motion discovery module to get a pre-trained model. In the second epoch, we first input batch data into the pre-trained model and get the corresponding probabilities {P r1 i , · · · , Prk i , · · · , Pr9 i } for each instance in V tr i . Prk i is the positive probability of the k-th instance in V tr i . When V tr i is a "fake" video, i.e., yi = 1, let yk i be 1 if Prk i is greater than 0.5, otherwise let yk i be 0. When V tr i is a "real" video, let yk i be equal to yi directly. We think this is more in line with the phenomenon that for a "fake" video, not all patches are easily recognized as "fake", while all local patches of a real one must be recognized as "real". Once we get the ground truth for each instance, we update the model parameters to get a new pre-training model. We continue the training process until the maximum number of iterations is reached.

In testing phase, for each video we will get 9 probabilities that indicate whether the video is "fake" or not. We pick the biggest instance probability as the final score of the video as a "fake" video. Experimental results show that our multiple instance learning framework can effectively improve the live face verification accuracy.

3.5 Implementation Details

In our experiments, we use Stochastic Gradient Descent (SGD) to optimize our model. 8 videos are randomly chosen and adjacent two frames are sampled from each video to form a batch. The ResNet in our model is initialized from a pre-trained model for ImageNet [Deng et al., 2009] classification. We start training with a learning rate of 0.005 and a weight decay of 0.001, and decrease the learning rate by 1/10 when the loss goes steady. All training and testing codes are built on Pytorch [Paszke et al., 2017]

4 Experiments

Live face verification must be robust across different types of attacks. We evaluate our proposed approach on the OULUNPU database [Boulkenafet et al., 2017] and the Replay Attack database [Chingovska et al., ].

4.1 Databases

The OULU-NPU database contains 990 real face videos and 3960 attack face videos. And it is divided into training, development and testing subsets. It claims that the models should be learned on the training subset, fine-tuned on the development subset, and tested on the testing subset. Further, it designs 4 protocols to evaluate the generalization of the live face verification methods by controlling the illumination, background scenes and the sensor devices of videos. For instance, in Protocol IV, models should be evaluated across unseen illumination, background conditions and sensor devices in the training and fine tuning phases. The performance metrics are the Attack Presentation Classification Error Rate (APCER) and the Bona Fide Presentation Classification Error Rate (BPCER).

The Replay Attack database contains 200 real face videos and 1000 attack videos. It is divided into training, development and testing subsets. And it follows the same training and evaluating rules as the OULU-NPU database. The performance metric is the Half Total Error Rate(HTER).

4.2 Results and Analysis

Here, we evaluate our model on the OULU-NPU and the Replay Attack database. Results are listed in Table 1. It shows that our model yields 1.8% HTER on the Replay-attack database and 4.6% ACER in Protocol I, 5.4% ACER in Protocol II, 4.0 % ACER in Protocol III, and 12% ACER in Protocol IV on the OULU-NPU database, which outperforms the previous method i.e. LBP [Boulkenafet et al., 2015].

To evaluate the effectiveness of each part of our model, we train a simple two-class classification network without motion map and multiple instance learning as our baseline. It yields 5.1% HTER on the Replay-attack database and 18.0% ACER on Protocol IV, OULU-NPU.

To demonstrate the effectiveness of motion map, we embed an STN network, which produce motion map from two adjacent frames, into the classification network. As we can see, CNN+STN, which yields 5.6% HTER on the Replayattack database and 18.3% ACER on Protocol IV, OULUNPU, performs much better than the baseline model(CNN). Moreover, we train the classification network with multiple instance learning. This CNN+MIL model yields 4.5% HTER on the Replay-attack database and 18.8% ACER on Protocol IV, OULU-NPU. Full results are listed in Table 1.

4.3 Generalization Analysis

We conduct a cross-database evaluation experiment to test the generalization capability of our proposed algorithm. All methods are evaluated on one database, while trained and fine-tuned on the other database.

It can be observed from table 2 that in terms of the ACER, our model gets the best generalization capability.

4.4 Hyper Parameters Analysis

We design an experiment to demonstrate why we divide face frames into 9 patches. In this setting, face frames are divided into average 1, 4, 9, 16 patches and tested on both data sets.

As is shown in Figure 4, the ACER (%) on both databases decreases as the number of patches increases. However, when the number of patches goes greater than 9, the ACER (%) does not increase obviously. When the patch numbers are set to 9, the ACER (%) reach 12% and 1.8% on the OULUNPU database and the Replay-Attack database, respectively. Therefore, we set the number of patches to 9, which is a good trade-off between effectiveness and efficiency.

4.5 Time Complexity Analysis

To estimate whether the proposed method can response quickly in practical application scenarios, we test the execution time of the LBP and our model, which are trained on the OULU-NPU database. Our set up consists of a intel i5-5350 CPU, a 8GB Memory and a Logitech HD C920 Pro webcam. The camera captures 50 real face videos and 50 replay attack videos to do live face verification. Table 3 shows, in terms of the execution time, our method is comparable to LBP. Our model shows superior APCER and BPCER performance in practical application scenarios.

5 Conclusion

In this paper, we present an accurate and robust method to tackle the live face verification task. The benefits of our proposed method are two-fold: 1) the Local Motion Discovery Module contributes to the verification accuracy, and 2) the Multi-Patch Examination Module enhances the recall rate of the attack videos. Extensive experimental results demonstrate the effectiveness of our proposed method.

Acknowledgments

This work was supported by National Science Foundation of China (U1611461,61502301,61521062). The work was partially supported by China's Thousand Youth Talents Plan, STCSM 17511105401 and 18DZ2270700.