Unsupervised Disentangled Representation Learning with Analogical Relations

1 Introduction

This paper is concerned with the problem of unsupervised disentangled representation learning in the generative model. The disentangled representation is a kind of distributed feature representation in which disjoint dimensions of a latent code reflect different high-level generative factors of data. Specifically, the disentangled representation can separate the explanatory factors which interact nonlinearly in the real-world data, such as the object shape, the material property, and the light source, etc. Thus, the disentangled representation is helpful for a large variety of AI tasks [Bengio et al., 2013].

The generative model is helpful in learning the disentangled representation. It is a methodology to learn a probability distribution, and it generates a new sample according to the code in the hidden space. By learning the appropriate parameter, it can gradually learn to generate new data of the same distribution as the target one [Goodfellow et al., 2014].

∗Corresponding author

When the disentangled representation is learned in the generative model, disjoint dimensions of a hidden code could model the data generative factors separately. These underlying factors could explain the major variation in the data. When only one factor varies but all others are fixed, the generated sequence of samples can show an interpretable change to human beings. For example, when we generate a figure of a hand-written digital number, a component of the code may be associated with the stroke width. When the value is changed, the stroke width of the generated number becomes smaller. A large body of work has been devoted to this problem. When the generative factors are predefined, the disentangled representation can be learned by the reconstruction of the data when the codes are swapped [Peng et al., 2017; Denton and Birodkar, 2017]. Moreover, when the data are labeled with attributes, the representation can be learned via the mapping between the data and the attributes [Wang et al., 2017] or the consistency between the variation of the data and the transformation of the latent code [Kulkarni et al., 2015; Worrall et al., 2017]. On the other hand, unsupervised learning of a disentangled representation has been a major challenge. DIP-VAE [Higgins et al., 2017] and β-VAE [Kumar et al., 2017] learn the disentanglement of the latent code by encouraging the latent distribution to be close to the standard normal distribution, in which each random variables are independent. InfoGAN [Chen et al., 2016] also uses the statistical independence and the method is motivated by the principle of the maximization of the mutual information. In summary, most of the existing works disentangle the factors by taking advantage of the supervised signals or by using the statistical independence of the prior distribution.

Different from the existing methods, our solution is motivated by the analogy. A key observation is that each interpretable disentangled factor is associated with an analogical relation of sample pairs. In the example of generating digital numbers, disjoint components of a hidden code can be associated with the factors such as rotation, stroke thickness, width, etc. Figure 1 gives an illustrative example of analogical pairs, which manifests the factor of stroke thickness. Reversely, this factor of variation can be learned from the analogical sample pairs. The discussed relation is known as the proportional analogical relation, which has a general form (a : b :: c : d) where the pairs (a, b) and (c, d) have relational similarity, such as mammals : lungs :: fish : gills [Gust et al., 2008]. The analogical relation can be extended to a batch of n sample pairs, which have the form (a1 : b1 :: a2 : b2 :: . . . :: an : bn). Namely, the pairs of ai and bi for all i = 1, . . . , n share the same analogical relation. The analogy has been a central part of human intelligence and cognition, so from this perspective learning the interpretable factors via the analogical relation is close to the human-like cognition process.

Based on the observation above, we propose our analogical training strategy on top of the generative model. At the beginning of the training process, the generator gives an analogical sample pair according to the code pair in which a predefined component of the latent code is different while all the other configuration is fixed. Next, an extra classifier tries to identify the predefined analogical relation behind the generated analogical pair. Then, the classifier and the generator are trained together to allow the classifier to make the correct decision. This is a cooperative game. The generator should learn to generate sample pairs characterized by analogical relations which the classifier can capture. Figure 2 gives an illustration of the analogical training process. This is our main contribution.

The other contribution is our proposal of a disentanglement metric of the learned representation, called the subspace score. This metric is based on two assumptions inspired by the subspace learning methodology. The first assumption is that each kind of variation of the generated samples forms an affine subspace, and thus these subspaces are expected to be uncovered by the subspace clustering algorithm. Therefore, the disentanglement of the factors is approximated by the clustering performance. The second is that the union of these subspaces should be close to the majority of observed samples. This closeness is measured by the distance between the observed samples and the affine space spanned by the generated samples. Our proposed subspace score is the combination of these two measures. The subspace score does not require supervised information, and it is able to be applied to the real-world unlabeled dataset. To the best of our knowledge, this is the first time an unsupervised disentanglement metric is proposed.

The rest of the paper is organized as follows. We introduce our proposed analogical training strategy in Section 2 and the disentanglement metric in Section 3. Next, we review related works in Section 4. In Section 5 we demonstrate the experiment results and compare our method with other methods along the subspace score. We conclude the paper in Section 6. Our source code will be available on https://github.com/ZejianLi/analogical-training.

Figure 2: The analogical training process. We assume each code component represents a generative factor, and each generative factor has a unique analogical relation. The analogical relation ri of the factor is implied by a pair of codes which are different only in the ith component. In the left of the figure, the horizontal axis is the ith dimension, and the vertical dimension indicates all the other dimensions. The code pair c1 and c2 are different only in the ith one. To start the training process, the code pair is passed through the generator G, which generates the analogical sample pair x1 and x2. Then, the classifier R tries to identify the underlying relation ri. Finally, the generator and the classifier both update their parameters such that the analogical relation can be classified more accurately, and thus the ith component learns to reflect a meaningful factor gradually. The training process can be applied on a batch of code pairs, and the analogical sample pairs also share the relation ri.

2 Method

2.1 Preliminaries

In this part, we briefly review two generative models, Variational Auto-Encoder (VAE) [Kingma and Welling, 2013] and Generative Adversarial Network (GAN) [Goodfellow et al., 2014]. A generative model learns the target probability distribution by generating new samples whose distribution is close the target one. Formally, given a latent representation z ∈ Z sampled from a predefined distribution Pz, we can generate a new sample x ∈ X by sampling from Pθ(x | z). Here Pθ(x | z) is assumed to be deterministic and described by a function G: Z ↦→ X parameterized by θ, and thus x = G(z). By learning appropriate θ, the generated distribution Pθ can get closer to the ground truth distribution Pr.

VAE has emerged as a popular deep generative model. A key step in VAE is to reinterpret the log-likelihood of the observed sample x in the generated distribution Pθ as

Here qφ(z | x) is the variational posterior distribution with the parameter φ and KL(·‖·) is the Kullback-Leibler divergence between two distributions. L(θ, φ; x) is the evidence lower bound defined as

The first term is the expected log-likelihood to recover x, and the second term is the KL-divergence between the variational posterior distribution and the prior distribution. Since we always have log pθ(x) ≥ L(θ, φ; x), we can increase the loglikelihood of the data by maximizing the evidence lower bound. Hence, the optimization problem of the model is

Here we use the function Q parameterized by φ to describe qφ(z | x).

GAN is another framework to train generative models. It learns the real distribution by training the generator G to confuse an adversarial discriminator D. The discriminator D tries to distinguish the data generated from Pθ or sampled from Pr, while the generator G gradually learns to generate data that D cannot correctly classify. Formally, given a sample x, D(x) approximates the probability that x is sampled from Pr. The formulation is given as follows.

To learn the disentangled representation in GAN, we divide the latent representation to the continuous code c and the noise z. The latent code c tries to capture the disentangled representation while the noise z fits other details. Thus the generated sample is given by x = G(c, z) where c ∼ Pc and z ∼ Pz. We do not adopt this division in VAE, since we find it leads to a trivial solution. For consistency, we write x = G(c, z) in the rest of the paper.

2.2 Analogical Training Strategy

In this part, we describe our proposed analogical training strategy, which is to learn the disentangled representation of interpretable factors. Our proposed method is based on the observation that a disentangled generative factor is related to a unique analogical relation of sample pairs. As shown in Figure 1, we have two different digits, and by reducing their stroke width we have their twins. The relation of the first digit and its twin is similar to the relation between the other digit and its twin. Even though the configurations of other factors are different, the change of the given factor shows the variation of the same pattern. Thus, the stroke width factor is shown by the analogical relation. Reversely, given a generator, if the analogical relation behind different generated pairs can always be recognized, we believe the learned factor is disentangled. More detailedly, each component of the hidden code represents a generative factor, and the change of the generative factor can be reflected by the variation in samples. Sample pairs which are different only in a single generative factor are defined as analogical pairs, and together they show the analogical relation unique to the generative factor. The model learns the disentangled factor by trying to generate analogical pairs.

Formally, we define a classifier R to recognize the analogical relations. Given an analogical pair of samples x1 and x2, R identifies the generative factor r in which the two samples are different. This is to maximize the log-likelihood

in which r is the generative factor as a random variable of the category distribution over {1, . . . , k} when we have k factors. Ex1,x2|r is short for E(x1,x2)∼P(x1,x2|r). Particularly, R(r | x1, x2) is the probability that R believe x1 and x2 are different in the factor r, which means x1 and x2 show the analogical relation unique to r.

Since samples in an analogical pair are different in one factor, we can generate the pair according to two latent codes which are different in one component. We denote the code pair as c1 and c2 and thus x1 = G(c1, z) and x2 = G(c2, z). Thus, (5) can be rewritten as

The process of the analogical training is visualized in Figure 2. The expectation can be approximated with the Monte Carlo method.

Theoretically, K(G,R) is the lower bound of the mutual information I(r; x1, x2) between the generative factor and the sample pair. Formally, I(r; x1, x2) = H(r) -H(r | x1, x2), where the first term is the entropy of r, and the second term is the conditional entropy of r given x1 and x2. We have H(r) = log k and

The computation of the posterior distribution P(r | x1, x2) is intractable. We can define an auxiliary distribution R to infer a variational lower bound.

Since the KL-divergence is non-negative, we can have

Therefore, K(G,R) is the lower bound of I(r; x1, x2). The bound is tight when R(r | x1, x2) is close to P(r | x1, x2). In this case, maximizing K(G, R) is equivalent to maximizing the mutual information I(r; x1, x2). Intuitively, this means the generator should make an effort to give sample pairs whose inner difference can show the variation of the generative factor.

Combining (3) and (6), we have our proposed Analogical VAE (AnaVAE) learned via the optimization problem

where λ is the hyperparameter to control the effect of K(G, R). Similarly, the optimization problem of our Analogical GAN (AnaGAN) is

Empirically, we set λ to 1 by default. All the functions G, Q, D and R are modeled by neural networks.

3 Disentanglement Metric

In this part, we introduce our disentanglement metric. The metric is inspired by the subspace clustering algorithm [Elhamifar and Vidal, 2013]. We assume that sequences of samples of the same variation lie in a low-dimensional affine subspace. Then if the factors in the representation are disentangled, the subspaces of different variations are independent. 1 In this case, samples of the same variation can be grouped by the subspace clustering method. An exemplary illustration of subspaces of variations is given in Figure 3. Another assumption is that the observed samples should be close to the affine space spanned by the generated samples, because the generated samples should be indistinguishable to the real ones. Our metric is designed according to these two assumptions. Since these assumptions are not related to any supervised information, the subspace score can be applied to unlabeled datasets.

1 A set of k linear subspaces {Si ⊂ RD }ni=1 are independent if dim(⊕ki=1Si) = ∑ki=1 dim(Si), where ⊕ is the direct sum.

The subspace clustering algorithm can separate data according to the relation of linear combinations and thus recover the low-dimensional affine subspaces. Given our assumption above, the disentanglement of factors can be approximated by how well the generated clusters of different variations can be correctly separated by the subspace clustering method. To be more specific, given the learned generator factor ri, we generate a sequence of samples {xj | j = 1, . . . ,m} by varying the ith component of the code while all other configurations are fixed. Thus, this sequence shows the variation of ri. A sample cluster consists of several different sequences of the same variation, and k clusters of different variations are sampled and given to the subspace clustering algorithm. Formally, suppose we have Y = [Y1, . . . ,Yk]. For i ∈ {1, . . . , k}, Yi is a group of sample sequences and the samples in each sequence are different only in the generative factor ri. Then the coefficient matrix U is learned by

Here, ‖ · ‖F is the Frobenius norm, and R(U) is the regularized term with the parameter λ′ . In addition, diag(·) denotes the diagonal elements in the matrix. We use the Orthogonal Matching Pursuit method (OMP) [Tropp and Gilbert, 2007] to learn the coefficients, since we have the prior knowledge of the number of samples in every cluster, which is the product of the number of sample sequences in each cluster and the size of the sequence. Then the affinity matrix is defined as

|U|+|U|T, where |·| is the absolute value. Ideally, this matrix is block diagonal given that samples in different Yi's lie in independent affine subspaces. Finally, with the affinity matrix, the spectral clustering method is used to infer the clustering assignment ˆ C. The clustering performance is measured by the normalized mutual information NMI( ˆ C,C) where C is the ground truth division of Yi's in Y. The performance measures how well clusters of different variations can be separated, and thus it evaluates the disentanglement of factors. This is the first part of our measure.

The other part is the mean distance from the observed samples to the affine space spanned by generated samples. This measures the closeness described in the second assumption. Formally, given x as an observed sample, we have the distance from x to its projection on the affine space spanned by Y as

When all the data are normalized to have unit length, we have d(x, Y) ∈ [0, 1]. Given the observed set X with n samples, the averaged distance is ¯ d = 1n ∑x∈X d(x, Y). We use 1 - ¯ d to estimate the closeness.

Finally, our proposed subspace score is defined as

We set α as 0.5 by default. The higher value of the subspace score is, the better the model learns the disentangled representation and generates new samples.

One may argue that the subspaces given by the principal component analysis (PCA) would get the highest score, because the assumptions of the subspace score are intrinsically compatible with PCA. However, the subspace score is to measure the variations of the generated samples. PCA lacks the ability to generate new samples, and it is not guaranteed to learn a concordant variation in each independent subspace. Thus, we do not think it is appropriate to apply the proposed metric on PCA. A novel disentanglement metric is also proposed in [Higgins et al., 2017], which can measure the independence and interpretability of factors simultaneously. However, this metric is applied on a synthetic dataset of 2D shapes with predefined factors, but not on the real-world dataset.

4 RelatedWorks

Several methods have been proposed for the unsupervised disentangled representation learning. InfoGAN [Chen et al., 2016] designs an extra network to recover the code values from the generated samples to learn disentanglement. This is based on the principle of maximizing the mutual information, and thus InfoGAN shares the theoretical foundation with our strategy. The difference between our strategy and InfoGAN is that our methods recover the relative relation from samples. We believe that the relation between samples are more important than the code values, because the code values have little physical meaning. For example, we cannot infer how many pixels of the stroke width is of a generated digital number from the code. However, the comparison between the sample codes leads to the conclusion that one digit is bolder than the other. Thus, the analogical training of sample pairs is more conceptually straightforward.

Different from InfoGAN, β-VAE [Higgins et al., 2017] is a variant of VAE. It learns the factorized factors by putting more emphasis on the KL-divergence between the variational posterior and the standard normal prior at the cost of the reconstruction accuracy. To avoid the cost, DIP-VAE [Kumar et al., 2017] designs an extra regularization term which minimizes the KL-divergence between the expectation of the variation posterior over the data and the prior. The main difference between these methods and ours is the way to encourage disentanglement. The existing methods disentangle the factors by leveraging the statistical independence in the standard normal prior distribution. However, in a direct manner our analogical training strategy requires that the change of one factor does not result in the variations of the other factors in the generated samples. This constraint allows the model to learn the disentanglement of factors directly.

5 Experiments

In this section, we present the experiment results on five image datasets, including MNIST [LeCun et al., 1998], CelebA [Liu et al., 2015], Flower [Nilsback and Zisserman, 2008], CUB [Wah et al., 2011] and Chairs [Aubry et al., 2014]. Specifically, we compare our methods with other state-of-the-art methods along the subspace score.

5.1 Implementation Details

Implementation details in the experiments are summarized here. For AnaGAN, we use the network architecture of DCGAN [Radford et al., 2015]. We use the WGAN-GP loss [Gulrajani et al., 2017] instead of the original GAN loss and the parameter num critic is set as 3. At the first 100 epoch, we only optimize V (D,G), because at the beginning of the training G has not captured the distribution yet and fails to generate samples good enough, and R cannot learn useful relations from G. Both the noise and the code are sampled from the standard normal distribution. We use Adam optimizer [Kingma and Ba, 2014] with a learning rate of 0.00002 and a momentum of 0.5. The batch size is 32. AnaVAE shares most of the configuration in AnaGAN. The encoder network Q borrows the major structure of D in AnaGAN. The learning rate for the Adam optimizer is 0.0001. In the experiment on MNIST with labels, we combine all methods in the comparison with AC-GAN [Odena et al., 2016] to incorporate the label information.

In both AnaVAE and AnaGAN, R has the similar structure of D, but the numbers of feature mappings in convolutional layers are halved. When learning the analogical relation of the factor ri, we first sample two identical codes c1 and c2. Then we set c2i (the ith component of c2) as the c1i - v or c1i + v where v is sampled from the uniform distribution over [1, 2]. If we take c2i = c1i - v only, R may separate a generative factor into two symmetric ones whose variations are in the opposite directions. The noise z1 and z2 are identical. We add a dropout layer [Srivastava et al., 2014] after each nonlinear activation layer in R to avoid overfitting. Lastly, the proposed algorithms are implemented with PyTorch.

To compute the subspace score, a cluster of ten sample sequences is generated for each factor and each sequence has five samples. The sequence is generated by varying the corresponding component of the code from -2 to 2 with the interval 1 but keeping other components fixed. We compute the subspace score over five different sets of generated samples to get the average. We implement it with scikit-learn.

5.2 Comparison

Figure 4 and Figure 5 provide a qualitative comparison of our proposed methods with β-VAE and DIP-VAE as appropriate. The pictures are generated by varying a specific latent variable from -3 to 3 while others are fixed to zero, so the samples in the central column of each figure grid are the same. This guarantees that the effect of only one factor is investigated every time. Figure 4 shows the learned factors in the Flower dataset. The models learn the factors including the color temperature of the flower, the variation of the color from yellow to magenta, and the number of flowers automatically. The first two factors are concerned with the colors, while the last with the structure of the picture. The β-VAE and DIP-VAE tend to generate the blurry pictures, in which the petals and pistils are not distinguishable, while AnaVAE can generate flowers in a more clear fashion and AnaGAN can even generate flowers with detailed textures. Particularly, although all the methods learn the variations of the number of flowers in the data, β-VAE and DIP-VAE fail to represent the two flowers in one picture in a clean way, while our proposed method can clearly separate the two flowers. The color of the flowers here is entangled with the flower shapes, since we can seldom have a sunflower in magenta in reality.

Figure 5 shows the comparison in the CelebA dataset. The models are able to disentangle factors of emotion, gender, and azimuth of the faces. The β-VAE tends to give face in a blurry pattern, but it disentangles the factors in a relative explicit way. DIP-VAE and AnaVAE have similar clarity here, but the factors they learn are slightly entangled with other variations. Specifically, the factors of emotion they learn are related to the width of the face; while the faces become smiling from left to right, they also become fatter. At last, AnaGAN gives faces with much more details than the other methods, but it tends to entangle the factors with other variations. The first two factors are entangled with the hair color. Generally, our proposed methods show comparable performances with other methods.

Table 1 reports the subspace score of models. AnaVAE outperforms other methods in the experiments on most datasets, while DIP-VAE has the highest score on the MNIST dataset. VAE has a similar performance to DIP-VAE. It has been discussed in [Kumar et al., 2017] that VAE also has the ability to disentangle factors in the data. AnaGAN does not have high scores here. Since the subspace score uses the mean squared error to estimate reconstructions, it is hard for samples of AnaGAN with detailed textures to have high reconstruction accuracies. InfoGAN also suffers from the same difficulty. In summary, AnaVAE shows competitive performances as compared with other methods along the subspace score.

6 Conclusion

In this paper, we propose the analogical training strategy to learn the disentangled representation without supervision. Based on the observation that there exists an analogical relation unique to a generative factor, the proposed strategy learns the disentangled factors from the generated analogical sample pairs. The strategy designs a classifier to help the generator to generate sample pairs of consistent analogical relations. We also propose the subspace score to measure the disentanglement of the factors. The subspace score does not require label information in the dataset. Experiments show that our proposed methods can uncover interpretable factors in the data and give competitive performances as compared with other state-of-the-art algorithms. In the future work, we will explore different measures of disentanglement and the application of the disentangled representation in foundational AI problems, such as the conceptual space learning.

Acknowledgments

This work is funded by the National Natural Science Foundation of China (NSFC) under Grant NO. 61773336 and NO. 91748127. We would like to extend our gratitude to the data providers. Zejian Li would like to thank Wei Li, Chao Wang and Ting He in Zhejiang University for helpful comments.