Inhibition of Occluded Facial Regions for Distance-Based Face Recognition

1 Introduction

This work focuses on the design and implementation of a Case-Based Reasoning (CBR) system for efficient face recognition, with a special focus on robust face recognition under partial occlusion conditions 1 . Although the problem of face recognition has been extensively addressed in the available literature, most state-ot-the-art proposals impose a series of constraints that limit their applicability in real world scenarios where only a limited amount of computational power and training information is available. The CBR method proposed in this paper seeks to cover the full recognition process (i.e., face detection, normalization, and identity prediction). In addition, we focused on methods which are able to work under the constraints of low computational power and little training information. As opposed to other occlusion-robust face recognition systems, the proposed CBR framework does not make any assumption about the nature of occlusion that it will have to face at test time. We also studied the possible integration of an efficient dimensionality reduction method in the proposed framework to reduce the computational cost. The experimental results presented in this paper show that the proposed method outperforms traditional face recognition methods in the task of partially occluded face recognition.

1 This paper is an abridged version of a paper titled "A CBR System for Efficient Face Recognition Under Partial Occlusion", presented at the ICCBR-2017 conference

2 RelatedWork

Most occlusion-robust face recognition systems available in the literature include a prior step to identification where they determine which parts of the images are affected by occlusion. In particular, some studies have used manually annotated occluded/non-occluded facial image patches to explicitly train a classifier [Min et al., 2011]. However, this approach has the drawback of requiring occluded face images during the training stage. As a consequence, if the nature of occlusion changes, the accuracy of the occlusion detector might be affected. Using color-based segmentation methods to detect occluded facial regions has also been proposed in the literature [Jia and Martinez, 2008]. However, these methods are very sensitive to lighting conditions and make the unsafe assumption that the occlusion is not caused by artefacts with human-skin color. [Ekenel, 2009] holds the idea that most of the accuracy loss registered by face recognition systems when dealing with partially occluded images is due to alignment errors, rather than information corruption by the occlusion. To address this problem, he proposed a method which seeks to minimize the distance between each sample in the training set and a new observation by evaluating a number of different alignment variations. As a consequence, searching for the best alignment variation requires hundreds of comparisons for each training sample. Although this method achieved notable accuracy rates, the computational cost supposes a major drawback.

Nowadays, the state-of-the-art in one of the most widespread face recognition datasets, namely the LFW dataset, is mostly held by deep neural network models 2 . However, such models are not always applicable due to computational constraints. In scenarios with this limitation, LBP descriptor-based methods remain as one of the best options.

3 Proposed Framework

In this section, we describe in detail both the proposed CBR framework and the selected pre-processing steps needed for face recognition. At test time, when the system is presented with an image that contains a human face in it, the following processing stages are executed: (1) A region of interest is determined for the human face in the image; (2) the detected face is aligned; (3) the image is pose-normalized, rotating and scaling the face to a standard size and orientation; (4) the lighting conditions of the image are normalized; (5) a feature extraction method is applied; and finally (6) the proposed retrieval and reuse stages are executed to emit a prediction regarding the identity of the person in the original image.

2 http://vis-www.cs.umass.edu/lfw/results.html#UnrestrictedLb

3.1 Preprocessing

This section describes the successive preprocessing stages executed before the actual retrieval and reuse stages in the proposed CBR framework.

• Face detection. For our experiments, we used the HOG face detector provided by the Dlib C++ library 3 .
• Face alignment. Our framework uses the face alignment algorithm proposed by V. Kazemi in 2014 [Kazemi and Josephine, 2014]. In particular, for the experiments in this paper we used the pre-trained model provided by Dlib C++ 3 .
• Pose normalization. Previously detected facial keypoints are used to normalize the facial image by rotating and cropping it to display the aligned face in its center, in a vertical pose. In addition, the resulting image is resized to a standard size.
• Light normalization. Due to its simplicity, efficiency and good performance, Histogram Equalization (HE) was used to normalize the lighting conditions in all experiments in this paper.

3.2 Feature Extraction: Local Binary Patterns

In this paper, we focus on a specific family of feature descriptors known as Local Binary Patterns (LBP) [Ojala et al., 2002]. As described in the following sections, the local nature of this descriptor will allow us to maintain features from occluded regions isolated from those extracted from visible parts of the face. The LBP descriptor labels pixels in an image by considering value differences with their neighbors. This label is treated afterwards as a binary number. The notation LBPP,R is usually used to denote the LBP descriptor with P neighbors and a radio of value R. It has been proved that, using the LBP8,1 descriptor, almost 90% of the extracted labels are uniform (i.e., its binary representation contains two transitions at most) [Ojala et al., 2002]. For this reason, a variant of LBP was designed where non-uniform patterns are merged together in a single label. This variant of the descriptor is known as uniform LBP (LBP u P,R). Before training a classifier, the LBP representation is often refined by dividing the image in a number of blocks (arranged in a grid structure) and counting the number of occurrences of patterns in each block. After this, the corresponding histograms of each block are concatenated to form the final descriptor. This approach in known as Local binary pattern histograms (LBPH).

3 Dlib C++ web page: http://dlib.net

Figure 2: Differential distribution of minimum local distances for occluded and non-occluded blocks in the ARFace database.

3.3 Identification: Occlusion-Robust Retrieval

First, we introduce a method to detect partial occlusion in LBPH blocks. Conveniently, our method requires no a priori knowledge about the nature of occluded blocks. We define the minimum local distance for the histogram of an LBP block as the minimum squared Euclidean distance obtained when comparing this histogram with the LBP histograms corresponding to the same facial region in the descriptors stored in the Case-Base. Then, the only assumption made by our method is that minimum local distances of occluded blocks are usually larger than those of unoccluded blocks. To provide insight into the veracity of this assumption, we calculated the distribution of minimum local distances for occluded and unoccluded blocks in the ARFace database 4 ; the resulting distributions are shown in figure 2. Although some overlapping exists among the two distributions, it might be possible to define a conservative threshold to discard most occluded blocks. Formally, the Case-Base of our framework is defined as a set of identity label y and LBPH descriptor x pairs:

When an unlabeled image I is presented to the system, it is first transformed by the successive preprocessing steps defined in the previous section. Afterwards, the LBPH descriptor x ∈ Rd of image I is generated and the retrieval stage begins. Our proposed retrieval stage begins by computing the n × d/p local distance matrix L, where p is the size of each histogram concatenated to form the LBPH descriptors. Each entry Lij in this matrix corresponds to the local distance between the j-th histogram in x and the j-th histogram in the i-th descriptor in the Case-Base; formally:

Based on this matrix and the desired threshold value for occlusion detection, the retrieve stage computes an occlusion mask M ∈ {0, 1}d/p that determines which of the histograms that conform descriptor x are considered as occluded:

4 See section 4 for details about the evaluation database.

Using this occlusion mask, the retrieval stage of the proposed CBR framework finds the k most similar cases to x in the Case-Base, according to the following dissimilarity function:

Intuitively, this dissimilarity function corresponds to the squared Euclidean distance between the features in x and x(i) that do not come from occluded facial zones (as predicted in the previous step). In other words, the proposed similarity measure dynamically inhibits the use of corrupted features while retrieving the most relevant cases from the Case-Base.

Afterwards, the reuse stage analyzes the retrieved cases and their labels to emit a prediction regarding the identity associated to the new case. To this extent, we use the weighted voting scheme proposed in [Hechenbichler and Schliep, 2004] with the dissimilarity function of eq. 4.

Conveniently, the computations defined by equations 2 and 3 can be done at the cost of time O(n( dp · p+ dp )), which simplifies to O(nd) given that p > 1. Finding the most similar cases in the Case-Base according to equation 4 takes O(n· dp + kn) time; which can be simplified to O(nd) by considering hyperparameters p and k as constants. Finally, the remaining computations which correspond to the voting process have a complexity of O(k). Therefore, the computational complexity of the complete test stage is O(nd) + O(nd) + O(k), which simplifies to O(nd) given that n >> k. Hence, we can conclude that the scalability of the proposed retrieval and reuse stages is equivalent to that of classical nearest neighbour search methods.

In the context of the proposed CBR system, the revision should be carried out by a human expert who determines whether an image has been assigned the correct identity. In particular, retaining revised cases only involves storing their case representation into the Case-Base. In addition, this mechanism can also be applied to provide the CBR system with knowledge of previously unseen individuals, thus extending the number of possible identities predicted by the system.

3.4 Multi-scale Local Binary Pattern Histograms

Several studies have found that higher recognition accuracy rates can achieved by combining LBPH descriptors extracted form the same image at various scales [Chan et al., 2007][Chen et al., 2013]. Unfortunately, the computational costs derived from using such a high-dimensional feature descriptor suppose a serious problem. Apart from that, this image descriptor is perfectly compatible with the proposed method. The only requirement is that histograms corresponding to the same image region are placed next to each other when forming the final descriptor. Then, selecting the correct value for p, the corresponding histograms for a specific face region will be treated as a single occlusion unit (i.e., a set of features which our method considers as occluded or non-occluded as a whole).

Local Dimensionality Reduction With Random Projection

This subsection tries to address the problem of highdimensionality of multi-scale LBPH descriptors. In the literature, [Chen et al., 2013] proposed using an efficient dimensionality reduction algorithm to reduce the size of multi-scale LBP descriptors. However, this approach is not directly compatible with the method proposed in the previous section. The reason for this being that we need to keep features from different occlusion units (i.e., facial regions) isolated form each other, so we can later detect and inhibit features coming from occluded facial areas. To overcome this limitation, we propose performing dimensionality reduction at a local level. To this extent, the histograms extracted from a specific facial region (at various levels) are considered as a single occlusion unit. Then, the Random Projection [Achlioptas, 2001] (RP) algorithm is applied locally to each occlusion unit. As opposed to other dimensionality reduction methods, RP generates the projection matrix from a random distribution. As a consequence, the projection matrix is data-independent and cheap to build.

The main theoretical result behind RP is the JohnsonLindenstrauss (JL) lemma. This result guarantees that a set of points in a high dimensional space can be projected to a Euclidean space of much lower dimension while approximately preserving pairwise distances between points [Dasgupta and Gupta, 2003]. Formally, given 0 < ϵ < 1, a matrix X with n samples from Rp , and k > 4 · ln(n)/(ϵ2 /2 - ϵ3 /3) a linear function f : Rp →Rk exists such that:

In particular, the map f : Rp → Rk can be performed by multiplying data samples by a random projection matrix R whose elements are drawn from N(0, 1):

As previously said, in order to apply RP in the context of the proposed method, me must first ensure that histograms coming from the same face region are placed contiguously in the final descriptor (see Fig. 3). Afterwards, we can apply the RP method locally to each occlusion unit. Formally, let x ∈ Rd be a multi-scale LBPH descriptor where each occlusion unit consists of p features. The reduced version of descriptor x is computed as follows:

where || denotes vector concatenation. Note that, thanks to the JL-lemma, for a sufficiently large k value the result of applying the proposed retrieval and reuse stages over reduced descriptors is approximately the same as doing it over the original high-dimensional descriptors. To prove this, it suffices to consider the different computations carried out by the proposed retrieval and reuse stages. First, the local distance matrix is computed according to eq. 2. If we reduce both the new case x and the descriptors x(i) in the Case-Base as described in eq. 7, and set hyperparameter p to the new size of occlusion units (i.e., p = k), equation 2 can be rewritten as follows:

where x′ ∈ Rd′ . Then, applying the JL-lemma, for a sufficiently large k value we can ensure that:

In other words, the distortion induced in matrix L′ with respect to L is bounded. The following steps of the proposed method are based on L′ . Therefore, if the difference between L′ and L is small enough, the proposed retrieval and reuse stages will provide the same results when executed over the reduced descriptors. Section 4 reports on several experiments where the descriptors were reduced with this approach.

4 Experimental Results

We evaluated the proposed system over a database of facial images with different types of occlusion and using different image descriptors. In particular, the evaluation dataset is the ARFace database [Martinez, 1998]. This dataset contains about 4,000 color images corresponding to 126 individuals (70 men and 56 women). The images display a frontal view of individuals' faces with different facial expressions, illumination conditions and partial occlusions. We used the images in the ARFace dataset to create several subsets for our experiments. In particular, we arranged a training set, a validation set, and several test sets with different characteristics:

• Training set: one image per individual (neutral, uniform lighting, first session).
• Validation set: almost one image per individual 5 (neutral, uniform lighting, second session).

5 Second session images are not available for all individuals in the dataset.

• Light test set: almost four images per individual (neutral, illumination left/right, first and second sessions).
• Glasses test set: almost two images per individual (sunglasses, uniform lighting, first and second session).
• Scarf test set: almost two images per individual (scarf, uniform lighting, first and second session).

We evaluate the proposed method against other common classification methods used in the field of face recognition, namely Logistic Regression (LR), Support Vector Machine (SVM) and Naive Bayes (NB). In the case of the proposed method, p determines the size of the occlusion unit, and is fully determined by the parametrization of the LBP descriptor. The remaining hyperparameter is the threshold for occlusion detection. This hyper-parameter can be selected using a validation set of images without occlusion, even if such validation set contains less than one image per individual. In particular, the threshold is set to the minimum possible value which does not produce misclassification of non-occluded blocks in the validation set as occluded (see [L´opez-S´anchez et al., 2017] for more details).

The evaluation protocol for all our experiments has been the following: (1) the classifier under evaluation is trained over the training set; (2) the validation set is used to perform hyperparameter selection; (3) the classifier, parametrized as determined in the previous step, is re-trained over the union of the training set and the validation set; (4) the trained classifier is evaluated over the different test datasets available.

Table 1 presents the results obtained by using the automatic face detection and alignment methods explained in section 3.1. Similarly, Table 2 compiles the results obtained by using the manual face alignment annotations provided by the ARface database. For single scale descriptors, we used LBP u 8,2 histograms over a 8 × 8 grid, thus obtaining a descriptor of 3, 776 dimensions. In the case of multi-scale descriptors, we used LBP u 8,2 histograms over 12 × 12 and 6 × 6 grids. The resulting descriptor dimension was therefore 10, 620. Finally, for our experiments with local RP (see Sec. 3.4), each 295dimensional occlusion unit in the high-dimensional multiscale descriptor was reduced to 150 features. Therefore, the complete descriptor ended up having a dimension of 5, 400 (i.e., approximately half the original dimension).

5 Discussion and FutureWork

This work proposed a novel CBR framework for occlusionrobust face detection. The retrieval and reuse stages of the system use a modified version of the weighted k-Nearest Neighbour [Hechenbichler and Schliep, 2004] algorithm to dynamically inhibit features from occluded face regions. This is achieved by using a novel similarity function which discards local distances imputable to occluded facial regions. As opposed to recent deep learning-based methods, the proposed system can operate in domains where only a small amount of training information is available, and does not require any specialized computing hardware to run. In addition, we proved that the Random Projection algorithm can be applied in a local manner to reduce the dimension of multi-level LBPH descriptors, while ensuring that the proposed retrieval and reuse stages will perform approximately as well as they do over the original high-dimensional descriptors. Experimental results carried out over the ARFace database show that, in most cases, the proposed method outperforms classic classification algorithms when using LBPH features to identify facial images with partial occlusion. As for future work, we propose investigating how automatic face alignment methods can be made more robust to partial facial occlusion.