Replicating Active Appearance Model by Generator Network

1 Introduction

Recently, a paper published in Cell [Chang and Tsao, 2017] reports an interesting discovery about the neurons in the middle lateral (ML)/middle fundus (MF) and anterior medial (AM) areas of the primate brain that are responsible for face recognition. Specifically, the paper is concerned with how these neurons respond to and encode the face stimuli generated by a pre-trained Active Appearance Model (AAM) [Cootes et al., 2001; 2015]. In AAM, there are explicit shape variables and appearance variables that generate the positions of the control points and the nominal face image respectively, and the output image is then generated by wrapping the nominal face image using the control points. [Chang and Tsao,

2017] discovers that the responses of the aforementioned neurons to the face image generated by the AAM exhibits strong linear relationship with the shape and appearance variables of the AAM that generates the face image. In fact, the shape and appearance variables of the AAM can be recovered from the neuron responses so that the face image can be reconstructed by the AAM using the recovered shape and appearance variables.

In this paper, we investigate whether the above phenomenon can be replicated by deep generativemodels. In particular, we focus on a popular deep generative model called the generator network [Goodfellow et al., 2014], which can be considered a non-linear generalization of the factor analysis model. Recall that in the factor analysis model, the signal is generated by latent factors that are assumed to be independent Gaussian random variables, and the signal is a linear transformation of the latent variables (plus observational noises). In the generator network, the latent variables still follow a simple known prior distribution such as independent Gaussian or uniform distribution, but the mapping from the latent variables to the observed signal is modeled by a convolutional neural network (ConvNet), which has proven to be an exceedingly powerful approximator of high-dimensional non-linear mappings.

Both the AAM and the generator network are latent variable models where the signal is obtained by transforming the latent variables. In the AAM, the latent variables consist of explicit shape variables and appearance variables, which generate the control points and the appearance image by linear mappings learned by principal component analysis (PCA). The output image is generated by a highly non-linear but known warping function of the control points and the nominal image. In contrast, the generator network is more generic, in that it does not assume any prior knowledge about shape and deformation, and it does not have any explicit shape variables and shape model. We are interested in whether the generator model can replicate the AAM in the sense that whether the generator network can learn from the images generated by a pre-trained AAM, so that the latent variables of the learned generator network are closely related to the latent shape and appearance variables of the AAM, and the non-linear mapping from the latent variables to the output image in the generator network accounts for the highly non-linear warping function of the AAM. As it is impossible for the latent variables of the learned generator network to be the same as the latent variables of the AAM, a strong linear relationship between the two sets of latent variables (or codes) is the best we can hope for. We shall show that such a linear relationship indeed exists, thus qualitatively reproducing the behavior of the neuron responses (or neural code) observed by [Chang and Tsao, 2017].

The generator network can be trained by various methods, including the wake-sleep algorithm [Hinton et al., 1995], variational autoencoder (VAE) [Kingma and Welling, 2013; Rezende et al., 2014; Salimans et al., 2015], generative adversarial networks (GAN) [Goodfellow et al., 2014; Radford et al., 2015; Denton et al., 2015], moment matching networks [Li et al., 2015], alternating back-propagation (ABP) [Han et al., 2017], and other related methods [Oord et al., 2016; Dinh et al., 2016]. They have led to impressive results in a wide range of applications, such as image/video synthesis [Dosovitskiy et al., 2015], disentangled feature learning [Chen et al., 2016] and pattern completion [Han et al., 2017] etc.

In this paper, we shall adopt the VAE method to train the generator network. Unlike GAN, the VAE complements the generator network with an inference network that transforms the observed image to the latent variables. The inference network seeks to approximate the posterior distribution of the latent variables given the observed image. The inference network and the generator network form an auto-encoder, where the inference network plays the role of the encoder that encodes the signal into the latent variables (or latent code), and the generator network plays the role of the decoder that decode the latent variables (or latent code) back to the signal. The parameters of the two networks can be learned by maximizing a variational lower bound of the log-likelihood [Blei et al., 2017]. We show that the latent variables computed by the inference network from the observed face image are highly correlated with the latent variables of the AAM that generates the face image.

Contributions. This paper is phenomenological in nature. It is our hope that the paper is of interest to both the neuroscience community and the deep learning community. The followings are the contributions of this paper:

• We study the linear relationship between the latent code learned by the generator network and the AAM code that generates the face stimuli. Our experiments suggest that the deep generative model exhibits similar behavior as the primate neural system.
• We show that the latent variables learned by the generator network can be separated into shape-related part and appearance-related part, and the generator network is expressive enough to replicate the AAM model.

2 Active Appearance Model (AAM)

The active appearance model [Cootes et al., 2001; 2015] is a generative model for representing face images. It has a shape model and an appearance model. Both models are learned by principal component analysis (PCA).

Shape model: The shape model is based on a set of landmarks or control points. In the training stage, the control points are given for each training image. Let x denote the coordinates of all the control points. The shape model is

where ¯x is the average shape, Ps is the matrix of eigenvectors, and bs is the vector of shape variables. (¯x, Ps) are shared across all the training examples, while x and bs are different for different examples. The model can be learned from the given control points of the training images by PCA, where the number of eigenvectors is determined empirically.

Appearance model: The appearance model generates the nominal image before shape deformation. To learn the model, we can wrap each training image to the shape-normalized image so that its control points match those of the mean shape ¯x. Then a PCA is performed on the shape-normalized training images. Let g denote the vector of the grey-level image. The appearance model is

where ¯ g is the mean normalized grey-level image, Pa is the matrix of eigenvectors, and ba is the vector of appearance variables. (¯ g, Pa) are shared by all the training examples, while g and ba are different for different examples.

We can learn (Ps, Pa) from the training images with given control points. We concatenate the shape and appearance variables to form the face representation or the latent code, i.e., ZAAM = [bs, ba]. Given ZAAM, we can generate face image Y by generating x and g first, and then warping g according to x using a warping function to output the image Y = h(g, x). The warping function h is given and is highly non-linear in g and x.

3 Generator Network

The generator network is a deep generative model of the following form:

where Z is the vector of latent variables (or latent code), d is the dimension of Z, i.e., the number of latent variables. Z is assumed to follow a simple prior distribution where each component is a Gaussian N(0, 1) random variable (Id is the d-dimensional identity matrix). The latent vector Z generates the output image Y by a non-linear mapping fθ, which is modeled by a top-down convolutional neural network (ConvNet), where θ collects all the weight and bias parameters of the top-down ConvNet. ϵ is the noise vector whose elements are independent N(0, σ2 ) random variables. Even though Z follows a simple distribution, the model can generate Y with very complex distribution and with very rich patterns because of the expressiveness of fθ. The generator model (3) is a generalization of the factor analysis model, where the mapping fromZ to Y is assumed linear.

Compared to the AAM, the generator network has no explicit shape model such as (1) with control points x and shape variables bs, nor does it have the explicit non-linear warping function Y = h(g, x). The generator network relies on the highly expressive ConvNet fθ to account for the linear shape model and the non-linear warping function. Even though no prior knowledge of shape and warping is built into the generator network, it can learn such knowledge by itself.

Specifically, we shall use a pre-trained AAM as a teacher model, and we let the generator network be the student model. The AAM generates training images, and the generator network learns from the training images. We shall show that the inferred Z from the face image Y has a strong linear relationship with the corresponding ZAAM that the AAM uses to generate Y .

4 Variational auto-encoder (VAE)

Given a set of N training images {Yi, i = 1, ...,N} generated by AAM, we train the generator network by variational auto-encoder (VAE) [Kingma and Welling, 2013; Rezende et al., 2014; Salimans et al., 2015]. Let p(Z) be the prior distribution of Z. Let pθ(Y |Z) be the conditional distribution of the image Y given the latent vector Z. Then the marginal distribution of Y is pθ(Y ) = ∫ pθ(Z, Y )dZ = ∫ p(Z)pθ(Y |Z)dZ. The log-likelihood is ∑Ni=1 log pθ(Yi), and in principle θ can be estimated by maximizing the loglikelihood. However, this is intractable because pθ(Z) involves intractable integral. The EM algorithm [Dempster et al., 1977] is also impractical because the posterior distribution pθ(Z|Y ) = p(Z)pθ(Y |Z)/pθ(Y ) is intractable. The basic idea of VAE is to approximate the posterior distribution pθ(Z|Y ) by a tractable inference model qφ(Z|Y ) with a separate set of parameters φ, such as a Gaussian distribution with independent components N(µφ(Y ), σ2φ (Y )), where µφ(Y ) is the vector of means of the components of Z, and σ2φ (Y ) is the vector of variances of the components of Z. Both µφ(Y ) and σφ(Y ) can be modeled by bottom-up ConvNets.

The parameters (θ, φ) can be learned by jointly maximizing the variational lower bound of the log-likelihood

where KL(q||p) denotes the Kullback-Leibler divergence from q to p. L(θ, φ) is computationally tractable as long as the inference model qφ(Z|Y ) is tractable. See [Kingma and Welling, 2013] for more details. qφ(Z|Y ) is the encoder, and pθ(Y |Z) is the decoder. After learning (θ, φ), we can estimate Z from Y by the learned posterior mean vector ZG = µφ(Y ). In our work, we use ZG as the code of Y .

We can understand VAE as follows. Let qdata(Y ) be the data distribution. Then the maximum likelihood is equivalent to minimizing KL(qdata(Y )||pθ(Y )) over θ. VAE is equivalent to minimizing

over both θ and φ, where qφ(Z, Y ) = qdata(Y )qφ(Z|Y ) and pθ(Z, Y ) = p(Z)pθ(Y |Z). Unlike the maximum likelihood objective function KL(qdata(Y )||pθ(Y )), which is the KL divergence between the marginal distributions, the variational objective function KL(qφ(Z, Y )||pθ(Z, Y )) is the KL divergence between the joint distributions. While the marginal distribution pθ(Y ) is intractable, the joint distribution pθ(Z, Y ) is tractable.

Define Q = {qφ(Z, Y ) = qdata(Y )qφ(Z|Y ), ∀φ} and P = {pθ(Z, Y ) = p(Z)pθ(Y |Z), ∀θ} be the two families of joint distributions. We can view VAE as the joint minimization of KL(q||p) over Q and P . Such joint minimization can be accomplished by alternating projection as illustrated by Figure 1, where Q and P are illustrated by two lines, and each distribution in Q and P is illustrated by a point. Starting from p = p0 ∈ P , we project p0 onto Q by minimizing KL(q||p) over q ∈ Q to obtain q1. Then we project q = q1 onto P by minimizing KL(q||p) over p ∈ P to obtain p1, and so on. This process will converge to a local minimum of KL(q||p). In Figure 1, the two projections are illustrated by two different colors, because they are of different natures. minq∈Q(q|p) is a variational projection that minimizes over the first argument, while minp∈P (q|p) is a model fitting projection that minimizes over the second argument. As is commonly known, the former has mode seeking behavior while the latter has moment matching behavior.

A precursor to VAE is the wake-sleep algorithm [Hinton et al., 1995], which amounts to replacing minimizing KL(q||p) over q ∈ Q by minimizing KL(p||q) over q by switching the order of p and q. The minimization of KL(p||q) can be accomplished by generating data from p in the sleep phase and learn qφ from the generated data. Because of the switched order, the wake-sleep algorithm does not have a single objective function. However, in wake-sleep algorithm, both projections are of the model fitting type.

The generator network can also be trained by GAN [Goodfellow et al., 2014]. However, GAN does not have an inference model or an encoder, which is crucial for our work.

5 Experiments

We conduct experiments to investigate whether the generator network can replicate or imitate the AAM, where the AAM serves as the teacher model and the generator network plays the role of the student model. In the learning stage, the generator network only has access to the images generated by the AAM. It does not have access to the shape and appearance variables (latent code) used by the AAM to generate the images. After learning the generator network, we investigate the relationship between the latent code of the learned generator network and the latent code of the AAM.

5.1 Experiment Setting

Data Generation. We pre-train the AAM using approximately 200 frontal face images with given landmarks or control points. Coordinates of the landmarks are first averaged, then PCA is performed where the first 10 principal components (PCs) for shape (see Eqn 1) are retained. The landmarks of each training image are then smoothly morphed into the average shape, so that the resulting image only carries shapefree appearance information. Another PCA is then performed on the shape-normalized training images, where the first 10 PCs for appearance (see Eqn 2) are retained. This results in a 20-dimensional latent face space, where every point represents a face. In other words, every face has a corresponding 20-dimensional AAM code denoted as ZAAM , which encodes its shape and appearance variables.

To generate face stimuli for our experiments, we randomly generate 20, 000 face images from the above pre-trained AAM. Specifically, for each dimension of the latent code, we record the standard deviation of the training responses of that dimension, and sample the variable from the Gaussian distribution with the same standard deviation as the real training faces. After that, these sampled variables are combined with the learned eigenvectors Ps and Pa to generate the synthesized images. The obtained images are then used as our training data for the generator network. Figure 2 shows some examples of training images to pre-train the AAM, and the synthesized face images generated by the trained AAM.

VAE Training. The training images obtained above are scaled so that the intensities are within the range [-1, 1]. The training images are also re-sized to [64, 64] to ease the computation.

For the generator network, we adopt the structure similar to [Radford et al., 2015; Dosovitskiy et al., 2015]. The network consists of multiple deconvolution (a.k.a convolutiontranspose) layers interleaved with ReLU non-linearity and batch normalization [Ioffe and Szegedy, 2015]. Specifically, we learn a 5 layer top-down convNet. The first deconvolutional layer has 512 filters with kernel size 4 × 4 and stride 1. There are 256, 128, 64, 1 filters with kernel size 4 × 4 and stride 2 for the second, third, fourth and fifth deconvolution layers respectively. Each deconvolution layer is followed by ReLU non-linearity and batch normalization except the last deconvolution layer which is instead followed by the tanh non-linearity.

For the inference model or the encoder network of VAE, we utilize the mirror structure of the generator network (which is the decoder network) where we use convolutional layers instead of deconvolutional ones. Besides, we use the ReLU

with leaky factor 0.2 as our non-linearity. The mean and variance networks of the inference model share the same network structure except the top fully-connected layer. We also adopt the batch normalization in the inference model as in the generator network.

We tried different dimensionalities for the latent code Z, including 20, 100 and 200 dimensions. We used Adam optimizer [Kingma and Ba, 2014] with initial learning rate 0.0002 for 500 iterations. The outputs of the mean network of the inference model are used as the learned latent code and are denoted as ZG. Realistic synthesized images can be generated by the trained generator network. See Figure 3 for some examples.

We design four experiments to examine the relationship between the AAM code ZAAM for generating the face images and the code learned by the generator network, ZG.

5.2 Linear Relationship

[Chang and Tsao, 2017] discovered that if a neuron has rampshaped tuning to different facial features, then its neural response can be approximated by a linear combination of the facial features. That is, the neural code for face patches ML/MF and AM has linear relationship with the AAM code of the presented face stimuli. In our first experiment, we check the strength of linearity between the code learned by the generator network and the underlying AAM code.

The 20, 000 codes for AAM, i.e., ZAAM, are used to predict the corresponding 20, 000 codes learned by generator network, i.e., ZG, and vice versa. Specifically, we fit linear model A and linear model B respectively:

We also include interception terms in both models. The goodness of fit of the model is determined by the percentage of variance in data that is explained by the fitted linear model, i.e., the so-called R-square (R2 ):

where Zi is the given code for image i, ˆ Zi denotes the fitted value, and ¯ Z is the average of the code. Higher R2 indicates stronger linear strength.

The R2 values for different dimensionalities of ZG are shown in the first two rows of Table 1. In addition to the convolutional-based (Conv) structures of the generator network stated above, we also test the linear relationship using fully-connected (FC) structures. Specifically, we learn 4 FC

layers with hidden dimension 256, using ReLU nonlinearity for the decoder and Leaky ReLU with factor 0.2 for the encoder, and all the layers are followed by batch normalization except the last ones. The R2 values are reported in the last two rows of Table 1. It can be seen that both models show strong linear relations. This is non-trivial and surprising, because when presented with only synthesized face stimuli, the VAE training of the highly non-linear generator network [Montufar et al., 2014] can automatically learn the code that is linearly related to the underlying AAM code that generates the given face stimuli. That is, the learned generator network shares similar behavior as the face patch systems ML/MF and AM in the primate brain.

5.3 Decoding

As argued in [Chang and Tsao, 2017], we should be able to linearly decode the facial features from the neural responses if there is a linear relationship between them. If so, we can accurately predict what the primate brain sees by knowing only the neural responses of the brain. Knowing that our learned code of the generator network ZG shows strong linear relationship with the facial features ZAAM from the above experiment, we expect that our automatically learned code ZG can accurately predict the facial features ZAAM, which can then be used to reconstruct the input face image via the AAM. Therefore we further examine the decoding quality in this section.

To proceed, for training, we use 20, 000 ZAAM and ZG obtained during the learning process to fit model B as described above. Denote the estimated coefficients as B⋆ . To test the decoding quality, we carry out the following two steps: (1) randomly sample a new set of 2, 000 AAM generated face images, which are used as the testing set. Then use the trained encoder network, i.e., mean network, of VAE to get point estimate of latent code of the generator network, i.e., Z test G . (2) Use the optimal B⋆ to get the predicted AAM code:

The predicted AAM code is then projected onto the previously learned AAM eigenvectors Ps and Pa to get the reconstructed image.

Figure 4 shows some testing images and the reconstructed ones. It can be seen that the linear model between the learned code by VAE and the AAM code gives us high decoding quality. In this experiment as well as the subsequent experiments, we set the dimensionality of ZG to be 100. Other dimensionalities give similar results.

5.4 Shape/Appearance Separation

The latent code ZG learned by the deep generative model is mixed with shape and appearance information. It would be useful to separate the shape and appearance parts of ZG. In this experiment, we further identify the strengths of shape and appearance parts of the learned code ZG.

From the first two experiments, we show that ZG is linearly related to the AAM code ZAAM, which contains both the shape and appearance parts. We can identify these two parts by projecting each dimension of ZG onto the 10 dimensional shape code bs and the 10 dimensional appearance code ba. We can then obtain the relative R2 for each part. A higher R2 for one part indicates the stronger response for this part. Recall that ZAAM = [bs, ba]. We fit the linear models on the shape part bs and the appearance part ba respectively:

Figures 5 and 6 show the R2 values for each dimension of ZG. It shows that each dimension of ZG responds differently to shape and appearance. To further verify and visualize our analysis, we first choose three dimensions with the top R2 for shape and three dimensions with the top R2 for appearance. Then we visualize the generated images using the trained generator network by varying (±3 sd) the three chosen dimensions of the learned code while keeping the other three dimensions fixed. Figure 7 shows the result. It is clear that if we only vary the shape dimensions of the code (horizontally in the figure), the generated images mainly change their shapes while the appearances tend to remain similar. On the other hand, if we only vary the appearance dimensions of the code (vertically in the figure), the generated images mainly change their appearances instead of shapes.

5.5 Replicating AAMby Supervised Learning

So far the generator network learns from the AAM in the unsupervised manner, where the generator network only has access to the training images but not the latent code of the AAM. We now examine whether the generator network has enough expressive power to replicate the AAM in the supervised setting where we also provide the latent code of the AAM to the generator.

In this experiment, the 20, 000 synthesized face images and their AAM codes are given, and we use these pairs to learn the generator network. To be more specific, we first train the generator network using the 20, 000 images and their codes. Let us denote the trained generator as G⋆ . Then, we prepare a new set of 2, 000 synthesized images Ytest and their AAM code Z test AAM as our testing set. Z test AAM is then fed into G⋆ to get ˆ Ytest. If the generator network is capable of replicating the AAM, then ˆ Ytest should be close to Ytest, that is, the generated images by the trained generator network should be similar to the testing face images.

We use the same generator network structure as in the VAE training. We use the stochastic gradient descent (SGD) algorithm with momentum 0.5 to train the generator network for supervised learning. The learning rate is 0.0001 with 900 epochs. Figure 8 shows the ground-truth testing images generated by the AAM and the reconstructed images generated by the trained generator network. We also calculate the perpixel ℓ1 reconstruction error, which is 0.0113.

6 Conclusion

The recent work in neuroscience [Chang and Tsao, 2017] shows that the face images can be reconstructed using the cell responses from face patches ML/MF and AM. To investigate whether the widely used generator network has the similar property, we design and conduct experiments to examine the relationship between the AAM code that generates the face stimuli and the automatically learned code by the generator network. Through the linearity analysis and the decoding quality analysis, we find that the biological observations made in [Chang and Tsao, 2017] can be qualitatively reproduced by the generator network, i.e., the learned code shows a strong linear relationship with the AAM code. Additionally, we can also use this relationship to further separate the shape and appearance parts of the learned code. Again this is similar to the neural system as it is found that ML/MF and AM carry complementary information about shape and appearance. Furthermore, we show that the generator network is capable of replicating AAM and we demonstrate this through supervised learning.

In this paper, we distill the knowledge of a pre-trained AAM to the generator network. It will also be interesting to distill the knowledge of a learned generator network to an AAM in order to interpret the generator network. We leave it to future work.

Acknowledgments

The work is supported by DARPA SIMPLEX N6600115-C-4035, ONR MURI N00014-16-1-2007, DARPA ARO W911NF-16-1-0579, and DARPA N66001-17-2-4029. Part of the work was done while the first author was visiting Microsoft Research in Seattle. We thank Dr. Gang Hua for his help.