Disentangled Face Attribute Editing via Instance-Aware Latent Space Search

1 Introduction

The task of face attribute editing aims to alter a given face image towards a given attribute, such as age, gender, expression, and eyeglasses. A successful editing should not only output high-quality results with accurate target attribute, but also well preserve all other image content characterized by the complementary attributes. Face attribute editing has attracted much attention in recent years and numerous algorithms have been proposed [Shen and Liu, 2017; Choi et al., 2018; Zhang et al., 2018; Bahng et al., 2020; Awiszus et al., 2019; Gu et al., 2019]. Notwithstanding the promising results demonstrated by these methods, this task is still quite challenging due to the high output dimension and need for precisely disentangling factors of variation corresponding to different face attributes.

* Corresponding Author: fuying@bit.edu.cn.

† Work of JY was done in September 2020.

InterfaceGAN

Figure 1: Our goal is to change the primal attribute of a given face (e.g., age and gender here) while preserving other condition attributes (e.g., eyeglasses and expression). Despite the latent space directions of the primal and condition attributes in the InterfaceGAN method have been orthogonalized, changing primal attribute still leads to unwanted condition attribute change. Our IALS method can produce satisfactory disentangled editing results.

Face image synthesis has achieved tremendous success recently with Generative Adversarial Networks (GANs) [Goodfellow et al., 2014]. State-of-the-art GAN models [Karras et al., 2019; Karras et al., 2020] can generate high-fidelity face images from the learned latent space. This motivates several works [Shen et al., 2020; Voynov and Babenko, 2020; Shen and Zhou, 2020; H¨ark¨onen et al., 2020] to edit face attribute by reusing the knowledge learned by GAN models. Specifically, for each attribute, they search a corresponding direction in the GAN latent space, such that moving a latent code along this direction can lead to the desired change of this attribute in the generated images. Although the target attributes can be changed effectively by these methods, disentangled editing is still problematic. As illustrated in Figure 1, eyeglasses may appear when age is changed from young to old, despite the latent space direction of age have been made orthogonal to eyeglasses via projection [Shen et al., 2020]. There are at least two possible reasons for this issue: i) the distribution bias of GAN's training set (e.g., elder people tend to wear eyeglasses in FFHQ [Karras et al., 2019]), and ii) the attribute-level directions cannot handle complex attribute distributions and are not effective for attribute variation disentanglement.

In this paper, we propose a novel framework to search the semantic directions in GAN latent space for disentangled face attribute editing. Instead of naively using fixed, attribute-level directions, we opt for dynamically searching instance-aware directions, where instance refers to the input image to be edited. The intuition behind is that by leveraging the instance information, the complementary attributes of the instance can be explicitly and effectively preserved, leading to disentangled editing results. To render the directions instance-aware, we consider the instancespecific direction obtained by back-propagating the gradient of off-the-shelf attribute CNN classifiers on the input image, and introduce a control factor to balance the attributelevel and instance-specific direction components. We propose a Disentanglement-Transformation (DT) metric to quantitatively evaluate the editing results and select the control factor that leads to the highest DT metric.

We test our method on both GAN-generated images and real ones, the latter of which are achieved by GAN inversion [Abdal et al., 2019] and re-generation. Experiments show that our method can achieve high-quality disentangled face editing results, outperforming state-of-the-art methods on both GAN-generate and real-world images by a wide margin. In summary, our contributions include:

• We propose a novel face attribute editing framework that searches for instance-aware semantic direction in GAN latent space, which explicitly promotes attribute variation disentanglement;
• We propose a Disentanglement-Transformation (DT) metric to quantitatively evaluate the editing efficacy and optimize our algorithm by leveraging this metric;
• We achieve high-quality results on both GAN-generated and real images that significantly outperform existing methods.

2 RelatedWork

Face attribute editing aims to manipulate the interested face attribute while preserving the rest. To achieve this goal, previous methods often leverage the conditional GAN model [Mirza and Osindero, 2014; Odena et al., 2017].These methods usually design the loss function [Shen and Liu, 2017; Bahng et al., 2020; Zhu et al., 2017] or network architecture [Choi et al., 2018; Zhang et al., 2018; Liu et al., 2019; Lin et al., 2019; He et al., 2019] manually to improve the output quality. Recently, 3D prior (e.g. 3DMM [Blanz and Vetter, 1999]) is also introduced to encode the synthetic image [Deng et al., 2020; Tewari et al., 2020]. These methods can generate high-quality results, but the diversity of controllable attributes is limited by the 3D priors (e.g., it cannot well model and edit gender information). Other recent methods leverage a high-quality synthetic face image dataset for disentangled representation training [Kowalski et al., 2020].

Another set of works [Shen et al., 2020; Voynov and Babenko, 2020; Shen and Zhou, 2020; Plumerault et al., 2020; H¨ark¨onen et al., 2020; Goetschalckx et al., 2019] propose to edit face attribute by moving the latent code along a specific semantic direction in the latent space of well-trained unconditional GAN model [Karras et al., 2018; Karras et al., 2019; Karras et al., 2020; Goodfellow et al., 2014]. Shen et al. [Shen et al., 2020] learn an SVM boundary to separate the latent space into the opposite semantic label and output the normal vector of the SVM boundary as the semantic direction. H¨ark¨onen et al. [H¨ark¨onen et al., 2020] sample a collection of latent codes and perform PCA on them to find principle semantic directions. However, these methods search the semantic directions on the attribute level, which cannot handle complex attribute correlations. Our method dynamically searches instance-aware semantic direction, which is effective for attribute variation disentanglement. It can edit various face attributes and obtain high-quality results.

3 Method

This section introduces the novel face attribute editing framework proposed by this paper. We begin by introducing face attribute editing with GANs and briefly revisiting prior methods using attribute-level latent directions, after which we present our instance-aware direction search algorithm.

3.1 Semantic Direction for Attribute Editing

Given a pretrained generator G from a state-of-the-art GAN model, e.g., StyleGAN [Karras et al., 2019], which maps a latent vector z to a face image, attribute editing can be achieved by moving z along a certain direction in the latent space. For real images, one can first embed them into the latent space to obtain the latent vector z and then modify it. The key is to find suitable semantic directions for attribute editing.

Let A denote a collection of face attributes, e.g., age, gender, expression, and eyeglasses. For each attribute X ∈ A, existing methods seek for a direction dX corresponding to X. For example, the InterfaceGAN method [Shen et al., 2020] first generates a large corpus of images G(z) by randomly sampling z. Then it labels the attributes of these images using a set of CNN binary classifiers H(·). Finally, it trains a SVM to separate each attribute label in the GAN latent space using these samples and outputs the normal vector of the SVM boundary as the semantic direction dX for each attribute. To achieve disentangled editing, it proposes to edit the primal attribute A while preserving one condition attribute B (or more) via direction orthogonalization:

where < ·, · > denotes inner product. Directions with more than one condition attribute can be obtained similarly as discussed in [Shen et al., 2020].

It can be seen that the directions so-obtained are at attribute level. They are fixed for each attribute and are instanceagnostic. We instead propose to incorporate instance information into direction search for better editing performance.

3.2 Instance-Aware Semantic Direction Search

Our instance-aware semantic direction consists of two parts: instance-specific and attribute-level directions. Next we first introduce the two components and then describe how to combine them.

Instance-Specific Semantic Direction

The generator G maps a latent code z to an image and an attribute classifier H maps an image x ∈ X to an attribute label. We can bridge the GAN latent space and attribute space via compositing H and G, i.e. H(G(·)). For attribute X, we can search for the instance-specific semantic direction for instance z, denoted by dX(z), via minimizing the following loss:

where y is the target attribute label and L(·, ·) is the classification loss with binary cross entropy function

We simply use gradient descent to search for dX(z) in Eq. (2), where dX(z) is updated by using an incremental direction updating scheme. To streamline the presentation, more details of incremental update is deferred to a later section. We further normalize dX(z) as the final instancespecific semantic direction for z:

Eq. (4) shows that opposite directions can be obtained with y = 0 and 1, respectively.

Attribute-Level Semantic Direction

Attribute-level directions aggregate the information across all training instances thus have higher resistance to noise. Similar to previous method, we also leverage attribute-level directions for attribute editing and use the one computed by InterfaceGAN as the default option.

Here we propose another way to compute the attributelevel direction. The intuition is to bridge the gap between the local instance-level and the global attribute-level information via sampling and averaging. Specifically, we can first randomly sample a set of GAN latent codes z and generate face images by them accordingly. Then, we compute the instancespecific direction for each sample and each attribute according to Eq. (4). Finally, for each attribute, we can average the instance-specific directions from all samples as the attributelevel direction dX. We find that these dX lead to similar attribute editing quality compared to the counterparts computed by InterfaceGAN, but this approach is simpler and easier to implement and eliminates the need for training SVMs.

Instance-Aware Semantic Direction

Our instance-aware semantic direction is constructed by injecting instance-specific information into the direction search process. Specifically, we formulate it as the combination of the aforementioned attribute-level and instance-specific directions:

where λ ∈ [0, 1] is the control factor to balance these two components. For conditional attribute editing, we rewrite Eq. (1) with our instance-aware semantic direction as

where A and B are the primal and conditional attributes, respectively. We set two different values for λ1 and λ2 for these two attributes respectively, considering we might require a different quantity of instance information when editing or dis-

entangling face attributes. Note that Eq. (6) degenerates to Eq. (1) when λ1 = λ2 = 1. In practice, we first solve for the (λ1, λ2) pair which can produce best results on a sample set, after which we fix them for the facial attribute editing task. Next, we discuss how to solve the (λ1, λ2) pair.

As mentioned previously, a good attribute editing should i) transform the primal attribute A to target label accurately, and ii) preserve the condition attribute B as much as possible. According to these criteria, we propose to use a Disentanglement-Transformation (DT) curve to evaluate the editing results for a pair of primal and condition attributes on a set of samples. In a DT curve, the x-axis represents the transformation accuracy p, i.e., the ratio of samples for which the primal attribute has been transformed into target label correctly in the editing results. The y-axis represents the disentanglement accuracy q, i.e., the ratio of samples for which the condition attribute on the edited images is consistent with their original ones.

Specifically, we randomly sample a set of latent codes in the GAN latent space and obtain the corresponding images generated by G. Then we edit these images by changing the latent code z to z + k · n · ˆ dA|B(z), where k ∈ R is the step size and n ∈ N is the number of steps. The transformation and disentanglement accuracy (pn, qn) are evaluated by CNN classifiers for the attributes. By continuously changing n and evaluating (pn, qn), we obtain a DT curve as illustrated in Figure 3 (in practice, the DT curve of our method is obtained via varying the incremental updating steps; see next section).

With DT curves generated, we can further compute the AUC (Area Under Curve) of DT , and we choose the (λ1, λ2) pair that maximizes the average AUC for all possible primalcondition attribute pairs via:

where N is the normalization factor. We numerically estimate the continuous integral in Eq. (7) using quadrature.

3.3 Incremental Direction Updating

Here, we present an incremental instance-aware direction search scheme for our editing method. We alternate between searching for the semantic direction and updating the latent code (and the output image) . Given the current latent code z and generated image G(z), we search for the new direction ˆ d(z) and update the latent code as z′ = z + k · ˆ d(z). The whole process is illustrated in Figure 2.

4 Experiment

In this section, we evaluate our IALS method and compare it against the state-of-the-arts. Due to space limitation, more experimental results and discussions are included in the suppl. material.

4.1 Settings and Implementation Details

We consider five face attributes in our experiment, i.e., expression, age, gender, eyeglasses, and pose, as in [Shen et al., 2020]. We will focus on the former four attributes in our experiments as we find pose is well disentangled from other attributes in GAN latent space (similar observations are found in [Shen et al., 2020; Deng et al., 2020]). We test our framework on the W space of StyleGAN generator trained on the FFHQ [Karras et al., 2018] and CelebA-HQ [Karras et al., 2019] datasets. The attribute classifiers H(·) are ResNet-18 [He et al., 2016] networks trained on the CelebA dataset [Liu et al., 2015]. The DT metrics are computed by another set of attribute classifiers with ResNet-50 structure. We empirically set the step size of incremental updating in our method to 0.1 in the following experiments.

4.2 Ablation Study

In this part, we investigate the effect of direction control factor and incremental updating scheme in our IALS framework.

Control Factor Pairs

To study the behavior of different combination weights for attribute-level and instance-specific directions, and solve for control factors λ1 and λ2, we evaluate the DT metric with λ1 and λ2 evenly sampled in [0, 1] with a step size of 0.25 (hence 25 (λ1, λ2) pairs in total). For each (λ1, λ2) pair, we set nmax = 20 (i.e. sample 20 points on the DT curve) and k = 0.1 and adopt the trapezoidal quadrature formula to approximate the integral in AUC computation defined in Eq. (7).

Table 1 shows the average AUC of all DT curves with different primal-condition attribute combinations. We have the following three observations: i) using attribute-level direction alone (i.e., λ1 = λ2 = 1) is clearly non-optimal; ii) the attribute-level information is more important to edit primal attribute (λ1 ≥ 0.75); and iii) adding instance information to condition attribute direction search significantly improves the editing results (λ2 ≤ 0.75).

Discussion Table 1 shows a performance plateau for a range of λ's (λ1 ≥ 0.75, λ2 ≤ 0.75), indicating that our method is insensitive to parameter selection within a reasonable range. It also shows that adding moderate instance-level information for the condition attribute would significantly better than using attribute-level information alone (λ2 = 1), demonstrating the efficacy of our framework. In the following we simply use (λ1, λ2) = (0.75, 0) for our editing method.

Incremental Updating

We further study the efficacy of our incremental updating scheme for instance-aware direction search. Figure 4 shows a typical result when changing age from young to old while preserving the eyeglasses attribute of the original face image. It can be seen that the results are clearly inferior if we do not use incremental updating (i.e., we keep using the instanceaware direction obtained in the first iteration).

4.3 Comparison with the State-of-the-arts

We compare our IALS method with several state-of-the-art face attribute editing method proposed recently, including InterfaceGAN [Shen et al., 2020], GANSpace [H¨ark¨onen et al., 2020], and STGAN [Liu et al., 2019].

Both InterfaceGAN and GANSpace are methods based on GAN latent space search. InterfaceGAN is a supervised semantic direction search method as mentioned in the previous section, while GANSpace is an unsupervised one which performs PCA on the sampled latent codes to find principle semantic directions in the latent space. We assign the directions found by GANSpace to interpretable meanings following [Shen and Zhou, 2020]. The STGAN method is based on image generation using conditional GAN. It uses an encoderdecoder architecture with a well-designed selective transfer unit for attribute editing. For these methods, we use the code or pretrained models released by the authors for evaluation and comparison.

Qualitative Results

we compare our method with the competitors on synthetic images and real images respectively. We adopt the conditional manipulation setup [Shen et al., 2020] with one primal attribute to be changed and one condition attribute to be preserved for the semantic direction based methods, and directly send the source image to the conditional-GAN based method.

Some typical qualitative results on GAN-generated images are provided in Figure 5. In can be seen that as the degree of attribute change increases, STGAN generated blurry images while the semantic direction based methods outputted highfidelity results. Furthermore, our method can well preserve the condition attribute of the original image when editing the primal attribute while the other two semantic direction based competitors often fail. InterfaceGAN and GANSpace only use attribute-level direction while ignoring the instance information when facial editing. By contrast, our method combines instance-specific and attribute-level information which results in much better disentangled facial editing results. The results also demonstrate that our method outperforms stateof-the-art methods on GAN-generated images.

Figure 6 shows two typical examples to illustrate the attribute editing efficacy of different methods on real images 1 where the goal is to edit age while preserving the eyeglasses attribute. We find that STGAN outputted blurry results again. On the other hand, the condition attributes (i.e. eyeglasses) are changed by GANSpace and InterfaceGAN. In contrast, our method obtains superior results in terms of both image quality and attribute variation disentanglement. It produces high fidelity results with satisfactory attribute modification and preservation.

Quantitative Results

To further evaluate the performance of our IALS method, we conducted a user study. we recruited 100 people and presented them with 720 groups of data, with each group consisting of 5 images - the original face image and the facial editing results of our method and the other 3 competitors. Each person was randomly assigned 18 groups of data and asked to choose all of the results they are satisfied with according to three criteria: the result looks natural, the primal attribute is well changed, and condition attribute is well preserved. Note that we did not ask them to select the best result, as optimal facial editing results are often not unique.

1 For InterfaceGAN, GANSpace and our method, we firstly embed the given images into the W+ latent space of StyleGAN using [Abdal et al., 2019] and then edit them.

The results in Fig. 7 show that our method obtained the highest average satisfactory rate (69.66% for ours vs. 62.06% for InterfaceGAN, 35.44% for GANSpace and 6.48% for STGAN). It significantly outperforms the other methods on some challenging attribute pairs such as gender as the primal attribute and expression as the condition attribute (76.44% for ours vs. 51.92% for InterfaceGAN).

5 Conclusion

We have proposed a novel IALS framework to achieve disentangled face attribute editing with GAN latent space search. The key ingredient is the integration of attribute-level and instance-specific directions, which leads to accurate target attribute control and, more importantly, highly disentangled attribute editing. To better integrate the attribute-level and instance-specific directions, we introduce a DisentanglementTransformation (DT) metric to find suitable control factor. The experiments collectively show that our approach obtains significantly better results than previous methods.

Acknowledgements

This research was supported by the National Natural Science Foundation of China under Grants No. 61827901 and No. 62088101.