CR-GAN: Learning Complete Representations for Multi-view Generation

1 Introduction

Generating multi-view images from a single-view input is an interesting problem with broad applications in vision, graphics, and robotics. Yet, it is a challenging problem since 1) computers need to "imagine" what a given object would look like after a 3D rotation is applied; and 2) the multi-view generations should preserve the same "identity".

Generally speaking, previous solutions to this problem include model-driven synthesis [Blanz and Vetter, 1999], datadriven generation [Zhu et al., 2014; Yan et al., 2016], and a combination of the both [Zhu et al., 2016; Rezende et al., 2016]. Recently, generative adversarial networks (GANs) [Goodfellow et al., 2014] have shown impressive results in multi-view generation [Tran et al., 2017; Zhao et al., 2017].

These GAN-based methods usually have a single-pathway design: an encoder-decoder network is followed by a discriminator network. The encoder (E) maps input images into a latent space (Z), where the embeddings are first manipulated and then fed into the decoder (G) to generate novel views.

1 The code and pre-trained models are publicly available: https://github.com/bluer555/CR-GAN

However, our experiments indicate that this single-pathway design may have a severe issue: they can only learn "incomplete" representations, yielding limited generalization ability on "unseen" or unconstrained data. Take Fig. 1 as an example. During the training, the outputs of E constitute only a subspace of Z since we usually have a limited number of training samples. This would make G only "see" part of Z. During the testing, it is highly possible that E would map an "unseen" input outside the subspace. As a result, G may produce poor results due to the unexpected embedding.

To address this issue, we propose CR-GAN to learn Complete Representations for multi-view generation. The main idea is, in addition to the reconstruction path, we introduce another generation path to create view-specific images from embeddings that are randomly sampled from Z. Please refer to Fig. 2 for an illustration. The two paths share the same G. In other words, G learned in the generation path will guide the learning of both E and D in the reconstruction path, and vice versa. E is forced to be an inverse of G, yielding complete representations that would span the entire Z space. More importantly, the two-pathway learning can easily utilize both labeled and unlabeled data for self-supervised learning, which can largely enrich the Z space for natural generations.

To sum up, we have following contributions:

• To the best of our knowledge, we are the first to investigate "complete representations" of GAN models;
• We propose CR-GAN that can learn "complete" representations, using a two-pathway learning scheme;
• CR-GAN can leverage unlabeled data for self-supervised learning, yielding improved generation quality;
• CR-GAN can generate high-quality multi-view images from even "unseen" dataset in wild conditions.

2 RelatedWork

Generative Adversarial Networks (GANs). Goodfellow et al. [Goodfellow et al., 2014] introduced GAN to estimate target distribution via an adversarial process. Gulrajani et al. [Gulrajani et al., 2017] presented a more stable approach to enforce Lipschitz Constraint on Wasserstein GAN [Arjovsky et al., 2017]. AC-GAN et al. [Odena et al., 2017] extended the discriminator by containing an auxiliary decoder network to estimate class labels for the training data. BiGANs [Donahue et al., 2017; Dumoulin et al., 2017] try to learn an inverse mapping to project data back into the latent space. Our method can also find an inverse mapping, make a balanced minimax game when training data is limited.

Multi-view Synthesis. Hinton et al. [Hinton et al., 2011] introduced transforming auto-encoder to generate images with view variance. Yan et al. [Yan et al., 2016] proposed Perspective Transformer Nets to find the projection transformation. Zhou et al. [Zhou et al., 2016] propose to synthesize views by appearance flow. Very recently, GAN-based methods usually follow a single-pathway design: an encoder-decoder network [Peng et al., 2016] followed by a discriminator network. For example, to normalize the viewpoint, e.g. face frontalization, they either combine encoder-decoder with 3DMM [Blanz and Vetter, 1999] parameters [Yin et al., 2017], or use duplicates to predict global and local details [Huang et al., 2017]. DR-GAN [Tran et al., 2017] follows the single-pathway framework to learn identity features that are invariant to viewpoints. However, it may learn "incomplete" representations due to the single-pathway framework. In contrast, CR-GAN can learn complete representations using a two-pathway network, which guarantees high-quality generations even for "unseen" inputs.

Pose-Invariant Representation Learning. For representation learning [Li et al., 2016; Fan et al., 2016], early works may use Canonical Correlation Analysis to analyze the commonality among different pose subspaces [Hardoon et al., 2004; Peng et al., 2015]. Recently, deep learning based methods use synthesized images to disentangle pose and identity factors by cross-reconstruction [Zhu et al., 2014; Peng et al., 2017], or transfer information from pose variant inputs to a frontalized appearance [Zhu et al., 2013]. However, they usually use only labeled data, leading to a limited performance. We proposed a two-pathway network to leverage both labeled and unlabeled data for self-supervised learning, which can generate realistic images in challenging conditions.

3 ProposedMethod

3.1 A Toy Example of Incomplete Representations

A single-pathway network, i.e. an encoder-decoder network followed by a discriminator network, may have the issue of learning "incomplete" representations. As illustrated in Fig. 2 left, the encoder E and decoder G can "touch" only a subspace of Z since we usually have a limited number of training data. This would lead to a severe issue in testing when using "unseen" data as the input. It is highly possible that E may map the novel input out of the subspace, which inevitably leads to poor generations since G has never "seen" the embedding.

A toy example is used to explain this point. We use MultiPIE [Gross et al., 2010] to train a single-pathway network. As shown in the top of Fig. 1, the network can generate realistic results on Multi-PIE (the first row), as long as the input image is mapped into the learned subspace. However, when testing "unseen" images from IJB-A [Klare et al., 2015], the network may produce unsatisfactory results (the second row). In this case, the new image is mapped out of the learned subspace.

This fact motivates us to train E and G that can "cover" the whole Z space, so we can learn complete representations. We achieve this goal by introducing a separate generation path, where the generator focuses on mapping the entire Z space to high-quality images. Fig. 2 illustrates the comparison between the single-pathway and two-pathway networks. Please refer to Fig. 3 (d) for an overview of our approach.

3.2 Generation Path

The generation path trains generator G and discriminator D. Here the encoder E is not involved since G tries to generate from random noise. Given a view label v and random noise z, G aims to produce a realistic image G(v, z) under view v. D is trying to distinguish real data from G's output, which minimizes:

where Px is the data distribution and Pz is the noise uniform distribution, Pˆx is an interpolation between pairs of points sampled from data distribution and the generator distribution [Gulrajani et al., 2017]. G tries to fool D, it maximizes:

where (Dv(·),Ds(·)) = D(·) denotes pairwise outputs of the discriminator. Dv(·) estimates the probability of being a

Algorithm 1: Supervised training with two paths

Input: Sets of view labeled images X , max number of

steps T , and batch size m.

Output: Trained network E, G and D.

for t = 1 to T do

for i = 1 to m do

1. Sample z ∼ Pz and xi ∼ Px with vi;
2. ¯x←G(vi, z);
3. Update D by Eq. 1, and G by Eq. 2;
4. Sample xj ∼ Px with vj (where xj and xi share the same identity);
5. (¯v, ¯z) ←E(xi);
6. ˜xj ←G(vj, ¯z);
7. Update D by Eq. 3, and E by Eq. 4;

end

end

specific view, Ds(·) describes the image quality, i.e., how real the image is. Note that in Eq. 1, D learns how to estimate the correct view of a real image [Odena et al., 2017], while G tries to produce an image with that view in order to get a high score from D in Eq. 2.

3.3 Reconstruction Path

The reconstruction path trains E and D but keeping G fixed. E tries to reconstruct training samples, this would guarantee that E will be learned as an inverse of G, yielding complete representations in the latent embedding space.

The output of E should be identity-preserved so the multiview images will present the same identity. We propose a cross reconstruction task to make E disentangle the viewpoint from the identity. More specifically, we sample a real image pair (xi, xj) that share the same identity but different views vi and vj. The goal is to reconstruct xj from xi. To achieve this, E takes xi as input and outputs an identity-preserved representation ¯z together with the view estimation ¯v: (¯v, ¯z) = (Ev(xi),Ez(xi)) = E(xi). Note that ¯v is learned for further self-supervised training as shown in Sec. 3.4.

G takes ¯z and view vj as the input. As ¯z is expected to carry the identity information of this person, with view vj's help, G should produce ˜xj, the reconstruction of xj. D is trained to distinguish the fake image ˜xj from the real one xi. Thus D minimizes:

where ˜xj = G(vj,Ez(xi)). E helps G to generate high quality image with view vj , so E maximizes:

where L1 loss is utilized to enforce that ˜xj is the reconstruction of xj. Lv is the cross-entropy loss of estimated and ground truth views, to let E be a good view estimator.

The two-pathway network learns complete representations: First, in the generation path, G learns how to produce real

Algorithm 2: Self-supervised training with two paths

• Input: Sets of view labeled and unlabeled images X , max number of steps T , and batch size m. Output: Trained network E, G and D. Pre-train E, G and D according to Algorithm 1; for t = 1 to T do for i = 1 to m do Sample z ∼ Pz and x ∼ Px; if x is labeled then 1. xi ←x; 2. Get the label vi of xi; 3. Repeat the step 2 to 7 in Algorithm 1; else 4. (¯v, ¯z) ←E(x); 5. Compute ˆ v (the estimation of ¯v); 6. Update D by Eq. 5 and E by Eq. 6; 7. Update D by Eq. 7 and G by Eq. 8; end end end

images from any inputs in the latent space. Then, in the reconstruction path, G retains the generative ability since it keeps unchanged. The alternative training details of the two pathways are summarized in Algorithm 1.

3.4 Self-supervised Learning

Labeled datasets are usually limited and biased. For example, Multi-PIE [Gross et al., 2010] is collected in a constrained setting, while large-pose images in 300wLP [Zhu et al., 2016] are distorted. As a result, G would generate low-quality images since it has only "seen" poor and limited examples.

To solve this issue, we further improve the proposed CRGAN with self-supervised learning. The key idea is to use a pre-trained model to estimate viewpoints for unlabeled images. Accordingly, we modify the supervised training algorithm into two phases. In the first stage, we pre-train the network on labeled data to let E be a good view estimator. In the second stage, both labeled and unlabeled data are utilized to boost G. When an unlabeled image x is fed to the network, a view estimation ¯v is obtained by Ev(·). Denote ˆ v to be the closest one-hot vector of ¯v, in the reconstruction path, we let E minimize Lv(¯v, ˆ v) and then reconstruct x to itself. Similar to Eq. 3, D minimizes:

similar to Eq. 4, E maximizes:

In the generation path, we let ˆ v be the ground truth of x, and generate an image in view ˆ v. So similar to Eq. 1, D minimizes:

similar to Eq. 2, G maximizes:

Once we get the pre-trained model, the encoder E predicts the probabilities of the input image belonging to different views. We choose the view with the highest probability as the estimation. Our strategy is similar to RANSAC algorithm [Fischler and Bolles, 1981], where we treat the estimations with higher confidence as "inliers" and use them to make view estimation more accurate. We summarize the self-supervised training in Algorithm 2.

Compared with the single-pathway solution, the proposed two-pathway network boosts the self-supervised learning in two aspects: 1) it provides a better pre-trained model for viewpoint estimation as a byproduct; and 2) it guarantees that we can take full advantage of unlabeled data in training since CR-GAN learns complete representations.

3.5 Discussion

To highlight the novelty of our method, we compare CR-GAN with the following three GANs. In Fig. 3, we show their network structures as well as ours for visual comparison.

BiGAN [Donahue et al., 2017; Dumoulin et al., 2017] jointly learns a generation network G and an inference network E. The authors proved that E is an inverse network of G. However, in practice, BiGAN produces poor reconstructions due to finite data and limited network capacity. Instead, CR-GAN uses explicit reconstruction loss to solve this issue.

DR-GAN [Tran et al., 2017] also tries to learn an identity preserved representation to synthesize multi-view images. But we have two distinct differences. First, the output of its encoder, also acts as the decoder's input, completely depends on the training dataset. Therefore, it can not deal with new data. Instead, we use the generation path to make sure that the learning of our G is "complete". Second, we don't let D estimate the identity for training data, because we employ unlabeled dataset in self-supervised learning which has no identity information. The involvement of unlabeled dataset also makes our model more robust for "unseen" data.

TP-GAN [Huang et al., 2017] uses two pathway GANs for frontal view synthesis. Their framework is different from ours: First, they use two distinct encoder-decoder networks, while CR-GAN shares all modules in the two pathways. Besides, they use two pathways to capture global features and local details, while we focus on learning complete representations in multi-view generation.

4 Experiments

CR-GAN aims to learn complete representations in the embedding space. We achieve this goal by combining the twopathway architecture with self-supervised learning. We conduct experiments to evaluate these two contributions respectively. Then we compare our CR-GAN with DR-GAN [Tran et al., 2017], both the visual results and t-SNE visualization in the embedding space are shown. We also compare CR-GAN and BiGAN with an image reconstruction task.

4.1 Experimental Settings

Datasets. We evaluate CR-GAN on datasets with and without view labels. Multi-PIE [Gross et al., 2010] is a labeled dataset collected under constrained environment. We use 250 subjects from the first session with 9 poses within ±60◦ , 20 illuminations, and two expressions. The first 200 subjects are for training and the rest 50 for testing. 300wLP [Zhu et al., 2016] is augmented from 300W [Sagonas et al., 2013] by the face profiling approach [Zhu et al., 2016], which contains view labels as well. We employ images with yaw angles ranging from -60◦ to +60◦ , and discretize them into 9 intervals.

For evaluation on unlabeled datasets, we use CelebA [Liu et al., 2015] and IJB-A [Klare et al., 2015]. CelebA contains a large amount of celebrity images with unbalanced viewpoint distributions. Thus, we collect a subset of 72,000 images from it, which uniformly ranging from -60◦ to +60◦ . Notice that the view labels of the images in CelebA are only utilized to collect the subset, while no view or identity labels are employed in the training process. We also use IJB-A which contains 5,396 images for evaluation. This dataset is challenging, since there are extensive identity and pose variations.

Implementation Details. Our network implementation is modified from the residual networks in WGAN-GP [Gulrajani et al., 2017], where E shares a similar network structure with D. During training, we set v to be a one-hot vector with 9 dimensions and z ∈ [-1, 1]119 in the latent space. The batch size is 64. Adam optimizer [Kingma and Ba, 2015] is used with the learning rate of 0.0005 and momentum of [0, 0.9]. According to the setting of WGAN-GP, we let λ1 =

10, λ2 ∼ λ4 = 1, λ5 = 0.01. Moreover, all the networks are trained after 25 epochs in supervised learning; we train 10 more epochs in self-supervised learning.

4.2 Single-pathway vs. Two-pathway

We compare two-pathway network with the one using a single reconstruction path. All the networks are trained on Multi-PIE. When test with Muli-PIE, as shown in Fig. 4 (a), both models produce desirable results. In each view, facial attributes like glasses are kept well. However, single-pathway model gets unsatisfactory results on IJB-A, which is an "unseen" dataset. As shown in Fig. 4 (b), two-pathway model consistently produce natural images with more details and fewer artifacts. Instead, the single-pathway model cannot generate images with good quality. This result prove that our two-pathway network handles "unseen" data well by learning a complete representation in the embedding space.

4.3 Supervised vs. Self-supervised Learning

The two-pathway network is employed in the following evaluations. We use Multi-PIE and 300wLP in supervised learning. For self-supervised learning, in addition to the above datasets, CelebA is employed as well. Note that we don't use view or identity labels in CelebA during training.

Evaluation on CelebA. Fig. 5 shows the results on CelebA. In Fig. 5 (a), although the supervised model generates favorable results, there are artifacts in all views. As the supervised model is only trained on Multi-PIE and 300wLP, it is difficult to "approximate" the data in the wild. Instead, the selfsupervised model has learned a latent representation where richer features are embedded, so it generates more realistic results while the identities are well preserved. We can make the similar observation in Fig. 5 (b). The supervised model can only generate images that are similar to Multi-PIE, while the self-supervised model can generate novel identities. In Fig. 5 (c) and (d), the self-supervised model preserve identity and attributes in a better way than others.

Evaluation on IJB-A. Fig. 6 shows more results on IJB-A. We find that our self-supervised model successfully generalize what it has learned fromCelebA to IJB-A. Note that it is our self-supervised learning approach that makes it possible to train the network on unlabeled datasets.

4.4 Comparison with DR-GAN

Furthermore, we compare our self-supervised CR-GAN with DR-GAN [Tran et al., 2017]. We replace DC-GAN [Radford et al., 2016] network architecture used in DR-GAN with WGAN-GP for a fair comparison.

Evaluation on IJB-A. We show the results of DR-GAN and CR-GAN in Fig. 6 respectively. DR-GAN produces sharp images, but the facial identities are not well-kept. By contrast, in Fig. 6 (a) and (b), CR-GAN produces face images with similar identities. In all cases, DR-GAN fails to produce highquality images with large poses. Although not perfect enough, CR-GAN can synthesize reasonable profile images.

Identity Similarities on IJB-A. We generate 9 views for each image in IJB-A both using DR-GAN and CR-GAN. Then we obtain a 128-dim feature for each view by FaceNet [Schroff et al., 2015]. We evaluate the identity similarities between the real and generated images by feeding them to FaceNet. The squared L2 distances of the features directly corresponding to the face similarity: faces of the same subjects have small distances, while faces of different subjects have large distances. Table 1 shows the results of the average L2 distance of CR-GAN and DR-GAN in different datasets. Our method outperforms DR-GAN on all datasets, especially on IJB-A which contains unseen data. Fig. 7 shows the t-SNE visualization in the embedding space of DR-GAN and CRGAN respectively. For clarity, we only visualize 10 randomly selected subjects along with 9 generated views of each. Compared with DR-GAN, CR-GAN produces tighter clusterings: multi-view images of the same subject are embedded close to each other. It means the identities are better preserved.

Generative Ability. We utilize DR-GAN and CR-GAN to generate images from random noises. In Fig. 8, CR-GAN can produce images with different styles, while DR-GAN leads to blurry results. This is because the single-pathway generator of DR-GAN learns incomplete representations in the embedding space, which fails to handle random inputs. Instead, CR-GAN produces favorable results with complete embeddings.

4.5 Comparison with BiGAN

111 To compare our method with BiGAN, we qualitatively show the image reconstruction results of both methods on CelebA in Fig. 9. We can find that as demonstrated by [Donahue et al., 2017; Dumoulin et al., 2017], BiGAN cannot reconstruct the data correctly, while CR-GAN keeps identities well due to the explicit reconstruction loss we employed.

5 Conclusion

In this paper, we investigate learning "complete representations" of GAN models. We propose CR-GAN that uses a two-pathway framework to achieve the goal. Our method can leverage both labeled and unlabeled data for self-supervised learning, yielding high-quality multi-view image generations from even "unseen" data in wild conditions.

Acknowledgements

This work is partly supported by the Air Force Office of Scientific Research (AFOSR) under the Dynamic Data-Driven Application Systems program, NSF CISE program, and NSF grant CCF 1733843.