Bayesian Tensor Inference for Sketch-Based Facial Photo Hallucination

1 Introduction

Recently, machine learning becomes more and more popular applied to the computer vision community. Various applications and methods that learn low-level vision have been proposed in the classical literature [Freeman et al., 2000]. In this paper, we focus on a fascinating vision topic: automatic image sketching which automatically generates alike sketch images from photo images. Sketches are the simplest form of drawings because they consist of only drawing lines. The artists can distill the identifying characteristics of a photo and highlight themwith a small number of critical strokes.

Implementing the great idea of learning vision, successful image sketching techniques try to observe and learn from the artist's works, and hence generate vivid and expressive sketches. As a result, example-based methods are widely studied in the latest years. Given a set of training images and their associated sketches drawn by artists, it is of interest to generate a sketch image automatically from an input image with the help of machine learning techniques.

For the particular image class of human faces, the transform between photo-realistic faces and their associative sketch drawings has shown promising applications in the future. In recent years, some works have been done to address the issues of synthesizing sketches from photos [Chen et al., 2001; Tang and Wang, 2004; Liu et al., 2005a] and sketch-based face recognition [Tang and Wang, 2004; Liu et al., 2005a]. However, to the best of our knowledge, a more difficult issue - the backward transform from sketches to photos - has not been seriously addressed. In this paper, we develop a novel research topic: hallucinating photorealistic faces from sketches, which is termed as sketch-based facial photo hallucination. We design a new face hallucination technique to fulfill the intractable backward transform. We also consider the forward and backward transforms together in order to explore the inherent relation between the bidirectional transforms shown in Fig. 1.

Considering the complexity of image spaces and the conspicuous distinction between photos and sketches, global linear models such as [Tang and Wang, 2004; Liu et al., 2005a] tend to oversimplify this problem. Therefore, we try to extract the local relations by explicitly establishing the connection between two feature spaces formed by a patch-based tensor model. To hold the statistical dependencies between pairwise patches with two styles more precisely and flexibly, we present a Bayesian Tensor Inference approach that incorporates the advantages of multilinear analysis techniques based on tensor into Bayesian statistics.

The rest of this paper is organized as follows. In Section 2, we elicit a tensor model for a facial image ensemble. The detailed rationale and algorithm for Bayesian Tensor Inference are presented in Section 3. Experimental results are shown in Section 4 and conclusions are drawn in Section 5.

2 Tensor Model

Recently, multilinear algebra and tensor modeling have attracted considerable attention in both computer vision and computer graphics communities. Research efforts applying tensor cover a broad range of topics including face modeling and synthesis [Vasilescu and Terzopoulos, 2002; Wang and Ahuja, 2003; Vlasic et al., 2005], super-resolution [Liu et al., 2005b], etc.

Motivated by previous multilinear approaches, we make use of a novel tensor model to exclusively account for the representation of images with two styles: photo-style and sketch-style. As small image patches can account for highlevel statistics involved in images, we take patches as constitutive elements of the tensor model. Based on a corpus containing image patches with photo- and sketch-styles, we arrange all these patches into a high-order tensor which will suffice to encode the latent connection between the two styles.

2.1 TensorPatches

Based on the observation that both styles, i.e. photo- and sketch-styles, share some common characteristics and each style possesses its special traits, we assume the existence of decomposition of the patch feature space into the common variation space, which reflects the commonalities shared by both styles, and the special variation space. Relying on this rationale, we employ multilinear algebra to perform tensor decomposition on one large patch ensemble carrying two modalities.

Let us divide training pairwise images into overlapping small square patches which are assumed to be of the same size. m pairs of patches within a spatial neighborhood located in images are collected to form a high-order tensor. Following the non-parametric sampling scheme [Chen et al., 2001], we allow m to be smaller than the length of each patch feature vector d. Resulting from the confluence of three modes related to patch examples, patch styles and patch features, a 3-order tensor D ∈ ℜm×2×d is built by grouping pairwise patches pertaining to the same neighborhood.

This paper adopts the High-Order SVD [Lathauwer et al., 2000] or N-mode SVD [Vasilescu and Terzopoulos, 2002], both of which are the generalizations of SVD, to decompose the higher-order tensor D as follows 1

where C, known as the core tensor, governs the interaction between the mode matrices U1, · · · , UN . Mode-n matrix Un contains the orthonormal vectors spanning the column space of matrix D(n) resulting from mode-n flattening (unfolding) D. Defining a tensor T = C ×3 Ufeatures as TensorPatches, then we have

So far, we succeed in decomposing the two-modal patch feature space into the common variation space spanned by Upatches ∈ ℜm×m and the special variation space spanned by Ustyles ∈ ℜ2×2 .

For any patch whether it is photo- or sketch-style, its corresponding tensor representation is

where w contains patch parameters encoding the common characteristics of pairwise patches, and sk, capturing style parameters reflecting the special properties of style k (1 denotes photo-style, 2 denotes sketch-style), is the k-th row vector of Ustyles. WhenwT is the i-th rowvector ofUpatches, the tensor representation of the i-th training patch with k-th style is obtained. Its vector representation can be derived via mode1(or 2, 3) flattening the subtensor P, that is, (f1(P))T .

For a new patch pair (x, y), x represents the photo-style and y denotes the sketch-style. Their tensor representations can be learned in line with eq. (3) by

where both A1 = T ×2 sT1 and A2 = T ×2 sT2 are constant tensors. Mode-1 flattening eq. (4) results in

Ax ∈ ℜd×m and Ay ∈ ℜd×m are called inferring matrices. The shared parameter vector w, expected to be solved, exists in the common variation space. We solve it through mapping pairwise patches into the common variation space as follows

where Bx = (ATx Ax)-1ATx and By = (ATy Ay)-1ATy are called mapping matrices. Fig. 2 illustrates the relations among x, y, w.

When one of (x, y) is unknown, inferring the counterpart style from the known style of a patch is the objective of image sketching or hallucination. Combining eq. (5) and eq. (6), a coarse estimation for y can be derived as

1 Details for tensor decomposition can be found in [Lathauwer et al., 2000; Vasilescu and Terzopoulos, 2002], we will not elaborate.

which coincides with the path shown in Fig. 2: mapping the "Photo Patch Space" to the "Common Variation Space" from which inferring the "Sketch Patch Space" .

Importantly, the coarse estimation offered by eq. (7) is consistent with sketch synthesis methods [Tang and Wang, 2004][Liu et al., 2005a] under particular conditions. Considering (x, y) as the holistic photo-sketch image pair of faces, when Ax and Ay are sample matrices whose columns are vectors recording the training sketch and photo images, eq. (7) is equivalent to the eigentransformmethod [Tang and Wang, 2004] where w contains linear combination coefficients w.r.t. training faces. In the case that Ax contains K nearest neighbors of input photo patch x in the training set and Bx collects associative sketches of these neighbors, eq. (7) resembles the local geometry preserving method [Liu et al., 2005a]. Furthermore, eq. (7) can be thought as the unconstrained solution of LLE [Roweis and Saul, 2000] where w is the weighting vector for locally linear reconstruction.

2.2 TensorPatches for Facial Image Ensembles

Due to the structural characteristics of face images, we adopt a higher-order tensor rather than the 3-order tensor modeled on generic images. For the facial image ensemble, it necessitates modeling both global structures and local details of face images together.

Given the multi-factor (people, spatial positions of patches) and multi-modal (styles) natures of these patches, we develop a 4-order tensor model to explicitly account for both styles and constituent factors of facial patches. Suppose we have n pairs of face images available for training, of which each is divided into m overlapping square patches. With two styles, the length of each patch feature vector is d. Therefore, patches are influenced by four modes: people, patches, styles and features. A 4-order tensor D ∈ ℜn×m×2×d is naturally built by grouping all photo-sketch patches sampled from the training faces. Perform tensor decomposition on D

where the core tensor C governs the interaction between 4 modes encoded in 4 mode matrices: Upeople ∈ ℜn×n, Upositions ∈ ℜm×m, Ustyles ∈ ℜ2×2 and Ufeatures ∈

ℜd×d, and TensorPatches T is obtained by forming the product C ×4 Ufeatures.

Significantly, the two factors (people, position) are crucial to determine a concrete patch and encoded in row vector spaces of mode matrices Upeople and Upositions. To remove factor (people, position) redundancies and suppress the effects of noise, a truncation of the two mode matrices is needed. In this paper, we adopt the N-mode orthogonal iteration algorithm in [Vasilescu and Terzopoulos, 2003] to perform factor-specific dimensionality reduction, outputting converged truncated mode matrices ˆUpeople ∈ ℜn∗r1 (r1 < n), ˆUpositions ∈ ℜm∗r2 (r2 <m) alongwith the rank-reduced approximation of tensor D

In analogy to the last subsection, for a new patch pair (xj, yj) residing in the j-th spatial position of face images, their tensor representations can be derived with the similar form as eq. (4)

where vT j and sTk is the j-th and k-th row vector of the mode matrix ˆUpositions and ˆUstyles, respectively. Both A1j = ˆ T ×2 vT j ×3 sT1 and A2j = ˆ T ×2 vT j ×3 sT2 are constant tensors. The people parameter vector w ∈ ℜr1×1 maintains to be solved for new patch pairs.

Mode-1 flattening eq. (9) results in

where Ajx , Ajy ∈ ℜd×r1 are position-dependent inferring matrices. Note that the people parameter vector w is shared in all patch pairs (xj, yj)mj=1 appearing in the same person. Defining a concatenated photo feature vector Ix = [x1; · · · ; xm] ∈ ℜmd×1 whose sketch counterpart is Iy = [y1; · · · ; ym] ∈ ℜmd×1 and two enlarged md × r1 matrices Ax = [A1x; · · · ;Amx ] and Ay = [A1y; · · · ;Amy ], we have

So we can solve the parameter vector w by the least square method

where both ATx Ax and ATy Ay are invertible because md >> r1 which is always satisfied throughout this paper.

3 Bayesian Tensor Inference

The learned tensor representations eq. (9) model the latent connection between Ix and w, or Iy and w. The concluded relations eq. (5), (6), (11), and (12) originate from factor decomposition which is implemented through tensor decomposition. Although eq. (7) gives a direct inference from x to y, we lack analysis in statistical dependencies between x and y.

Utilizing the merits of super-resolution techniques [Freeman et al., 2000; Baker and Kanade, 2002], we incorporate the learned relations eq. (11) and (12) in which the tensor model entails into a Bayesian framework. The entire inference process is thus called the Bayesian Tensor Inference.

3.1 Formulation

We fulfill the backward transform Iy → Ix through the people parameter vector w by taking these quantities as a whole into a global optimization formulation. Our inference approach is still deduced from canonical Bayesian statistics, exploiting PCA to represent the photo feature vector Ix to be hallucinated. The advantage of our approach is to take into account the statistics between the latent variable a ∈ ℜl and Ix. We write the eigenface-domain representation after performing PCA on the training photo vectors {I(i) x }ni=1

where PCA noise ε may be ignored. The prior p(a) is easier to acquire

where Λ is a l × l diagonal matrix with eigenvalues at its diagonal.

Due to the Bayesian MAP (maximum a posterior) criterion, we find the MAP solution a∗ for hallucinating the optimal I∗x as follows

We take the statistics between w and Iy into consideration. Due to the learned relation (see eq.(11)) Iy = Ayw, we assume the representation error of Iy w.r.t w is i.i.d. and Gaussian. The conditional PDF p(Iy|w) is thus obtained as follows

where λ1 scales the variance in the photo space. In analogy to above, we obtain the condition PDF p(w|a) in doing w = BxIx and using eq. (13)

where λ2 scales the variance in common variation space.

Substituting PDFs given in eq. (14), (16), and (17) into eq. (15), maximizing p(w, a|Iy) is equivalent to minimizing the following object function

Significantly, function (18) is quadratic, and consequently minimizing it is a quadratic optimization problem (unconstrained) where a globally optimal solution exists.

Taking the derivative of E with respect to w and a respectively, the partial gradients of E can be calculated as

By ∂E/∂w = 0 and ∂E/∂a = 0, we obtain a pair of solutions {w∗ , a∗ } that is unique, and so must be globally optimal. The optimal MAP solution a∗ is hereby given as

where the r1 × r1 matrix C is predefined as

So far, we accomplish the task of backward transform, i.e. hallucinating the photorealistic photo feature vector I∗x = Ua∗ +μ given a sketch feature vector Iy based on an available set of myriad training photo-sketch image pairs.

3.2 Algorithm

Our algorithm tackles the intractable problem: sketch-based facial photo hallucination with a learning-based scheme. There have been several successful relevant efforts on facial sketch synthesis. Example-based sketch generation [Chen et al., 2001] takes pixels as rendering elements, so the computational efficiency is somewhat low. Linear methods [Tang and Wang, 2004; Liu et al., 2005a] tend to trivialize the synthesized results without any statistical consideration. The fact that our work improves upon the existing techniques is that patch-based tensor learning and Bayesian inference are coordinated in our algorithm, which is illustrated in Fig. 3.

Collect a large corpus of n training facial image pairs {I (n) x , I (n) y }Nn=1 with two styles, and divide each image into m overlapping patches. Because the sketch image space and photo image space are disparate in terms of gray-level intensities, to bring the two spaces into correspondence, we use the horizontal and vertical image derivatives as features of both photo- and sketch-style patches for learning the TensorPatches model. Concretely, we envision Bayesian Tensor Inference involving two phases: the training and testing phase.

Training Phase

1. Based on the derivative feature representation, build a 4order tensor on n ∗ m photo-sketch patch pairs sampled from n training image pairs and learn the matrices Ay and Bx.
2. Construct concatenated photo feature vectors {I(i) x }ni=1 , and run PCA on them to achieve the eigen-system (U, Λ) and the mean vector μ.

input image derivatives inferred image derivatives

Figure 3: Architecture of our sketch-based facial photo hallucination approach. (1) Learn the TensorPatches model taking image derivatives as features in the training phase; (2) obtain the initial result applying the local geometry preserving method; (3) infer image derivatives of the target photo using the Bayesian Tensor Inference method, given input image derivatives extracted from the test sketch face; (4) conduct gradient correction to hallucinate the final result.

Testing Phase

1. For a new sketch image Iy, divide it into m patches the same way as in the training phase. Using grayscale intensities for image features, apply the local geometry method [Liu et al., 2005a] to estimate an initial hallucinated result I 0 x .
2. Construct the concatenated sketch feature vector Iy using m overlapping patches, and exploit it and Ay, Bx, U, Λ, μ to infer the target photo feature vector I∗x according to eq.(19).
3. Break I∗x into m patches and afterward integrate these patches into two holistic derivative maps I h x and I v x , with pixels in the overlapping area blended.
4. Conduct gradient correction on I 0 x using the derivative maps I h x , I v x (i.e. gradient field) to render the final result I f x .

4 Experiments

Sketches often have many styles, for example, a cartoon sketch often represents a person very exaggeratedly. In this paper, we only focus on sketches of plain styles without exaggeration, as for applications in law enforcement or photo search, no much exaggeration is allowed in sketch images so that the sketches can realistically describe the real subjects. Some sketch examples are shown in Fig. 4 and Fig. 5.

We conduct experiments on a database over 600 persons of which 500 persons are for training and the rest is for testing. Each person has a frontal face photo image and a sketch image with a plain style drawn by an artist. All the training and testing images are normalized by affine transform to fix the position of centers of eyes and mouths. Each 160 × 100 face image is divided into 160 overlapping 13 × 13 patches in the same way, and the overlap between adjacent patches is three pixels.

To display the effectiveness of our Bayesian Tensor Inference, we compare it with the representative methods, which have been applied in facial sketch synthesis, including the global method- eigentransform [Tang and Wang, 2004] and the local method- local geometry preserving (LGP) [Liu et al., 2005a]. Fig. 4 and Fig. 5 illustrates the results of sketchbased face hallucination for Asians and Europeans, respectively. In both the two groups of results, patch-based methods (c), (d) consistently and notably outperform the global method (b) in high fidelity of local details, which indicates that a local model is more suitable for modeling complex distributions such as images. By comparing the results yielded by LGP (c) and those produced by our method (d), we can clearly see that our inference algorithm performs sharper and more realistic with higher image quality than LGP. One reason is that employing the patch-based tensor model can recover both the common face structure and local details.

We set the scale parameters λ1 and λ2 to be 0.02 and 0.2, respectively. We use standard SVD to do PCA on the training photo feature vectors, with 97% of the eigenvalue total retained. When applying the LGP method to obtain the initial hallucinated result in step 1 of the testing phase, we choose K = 5 to be the number of nearest neighbors.

From the quantitative perspective, Table 1 lists the average root-mean-square pixel errors (RMSE) for 4 different methods. The baseline is the nearest neighbor method. As might be expected, the performance of our method does improve as the number of training sample pairs increases from 250 (Training Set I) to 500 (Training Set II). Although the RMSE of our method is larger than that of LGP, the hallucinated images look better, as detailed as that in the groundtruth face images. To sum up, our sketch-based face hallucination method acquires the best visually perceptual quality.

5 Conclusions

In this paper, we present a statistical inference approach called Bayesian Tensor Inference between the photo patch space and the sketch patch space. Based on the principle that only commonalities contribute to the inference process and by virtue of the patch-based tensor model, we capture and learn the inter-space dependencies. The Bayesian MAP framework integrates tensor modeling and statistical optimization to ensure the inference performance of the difficult vision problem: sketch-based facial photo hallucination. The potency of our method is sufficiently shown in the comparative experiments, achieving surprising photo rendering quality.

Figure 5: Photo hallucination results for European faces. (a) Input sketch images, (b) eigentransform method, (c) local geometry preserving method, (d) our method, (e) groundtruth face photos.

Acknowledgement

The work described in this paper was fully supported by grants from the Research Grants Council of the Hong Kong Special Administrative Region and a joint grant (N CUHK409/03) from HKSAR RGC and China NSF. The work was done at the Chinese University of Hong Kong.