Joint Generative Moment-Matching Network for Learning Structural Latent Code

1 Introduction

In recent years, deep generative models have attracted much attention among machine learning and computer vision communities. Due to deep learning techniques' flexibility and powerful capability on fitting nonlinear functions, deep generative models have shown promising capabilities in characterizing the distribution of complex datasets.

Among various generative models, Generative Adversarial Network (GAN) [Goodfellow et al., 2014] has achieved much progress in many tasks, such as image generation [Goodfellow et al., 2014; Chen et al., 2016; Reed et al., 2016], image translation [Zhu et al., 2017; Kim et al., 2017]. Specifically, given a dataset X ∼ QX where QX might be very complicated, GAN learns a transformation gθ(·) by a deep neural network, which operates on a simple distribution Z ∼ PZ, such that Pgθ(Z) is a good approximation of the sample distribution QX. To measure the similarity between Pgθ(Z) and QX, GAN employs a min-max optimization schema where another deep neural network fϕ(·) is trained to measure the similarity of these two distributions. Although this schema has shown success in many tasks, yet it is difficult to optimize because it requires special techniques [Arjovsky et al., 2017; Gulrajani et al., 2017] to converge to a good solution. Instead, Generative-Moment Matching Network (GMMN) [Li et al., 2015] employs Maximum Mean Discrepancy (MMD) as the measurement for the similarity of two distributions, which is much simpler than the objective of GAN models and easy to optimize. Specifically, it employs the kernel embedding [Gretton et al., 2012] technique to match all orders of statistics between Pgθ(Z) and QX. In this paper, we will focus on the GMMN model.

Although the GMMN model can successfully learn the approximation of the underlying sample distribution QX, yet it lacks the inference capability, failing to learn the posterior distribution of the latent code. Since many downstream tasks heavily depend on the latent code, such as the classification and clustering, it is important and necessary to endow GMMN model with inference capability.

To address the inference problem, we propose a Joint Generative Moment Matching Network (JGMMN) as shown in Figure 1. Specifically, it includes two networks: the generation network and the inference network. The generation network works as a generative model gθ(·) to generate samples: ˆX = gθ(Z) ∼ PX|Z,Z ∼ PZ. The inference network defines an inference model hβ(·) to infer latent codes from observed samples: ˆ Z = hβ(X) ∼ QZ|X,X ∼ QX. Importantly, this problem can be formulated as the matching of two joint distributions QX,hβ(X) and Pgθ(Z),Z. If they are matched, corresponding marginal distributions and conditional distributions also can be guaranteed to match [Dumoulin et al., 2016]. Especially, the desired posterior distribution PZ|X can be matched by the conditional distribution QZ|X [Dumoulin et al., 2016].

However, it is challenging to match the two joint distributions Pgθ(Z),Z and QX,hβ(X). Since the marginal distributions QX and Qhβ(X) are not independent, we cannot match two marginal distributions (QX with Pgθ(Z) and Qhβ(X) with PZ) respectively. Thus, how to match the two joint distributions is challenging. Another challenge is that the learned QZ|X should be consistent with the sample distribution QX. Specifically, the inferred latent code distribution QZ|X should have similar structures with the sample distribution QX. More specifically, if the sample distribution QX is a multi-modal distribution, QZ|X should also be a multimodal distribution such that the inferred latent code is a good representation of observed samples. Then, the downstream tasks can benefit from it.

To address the first challenge, we propose to use the Joint Maximum Mean Discrepancy (JMMD) metric to measure the similarity between two joint distributions. In detail, JMMD measures the distance of two joint distributions by their kernel embeddings in a Reproducing Kernel Hilbert Space (RKHS). This objective can be efficiently optimized by back-propagation. By minimizing this objective, we can match Pgθ(Z),Z to QX,hβ(X), learning the generative model and the inference model simultaneously. To address the second challenge, we propose a novel multi-modal regularization to enforce the latent code distribution QZ|X to be consistent with the sample distribution QX. At last, extensive experiments are conducted to verify the effectiveness of our proposed model.

2 RelatedWorks

In this section, we will review the most related works, analyzing their properties and giving the motivation of our proposed JGMMN.

In recent years, much progress has gone towards deep generative models for modeling complicated data distributions. Among them, Generative Adversarial Network (GAN) [Goodfellow et al., 2014] has attracted much more attention in computer vision and machine learning communities, and has achieved impressive results in many tasks, such as image generation [Goodfellow et al., 2014; Chen et al., 2016; Reed et al., 2016; Ledig et al., 2016; Radford et al., 2015;

Denton et al., 2015], image translation [Zhu et al., 2017; Kim et al., 2017]. Generally, GAN trains a generator, which transforms a simple latent distribution to a complicated sample distribution. Meanwhile, it also trains a discriminator, which distinguishes the generated and real samples. Mathematically, GAN is to optimize a min-max problem. However, this optimization problem is difficult to optimize, which usually needs special techniques [Arjovsky et al., 2017; Gulrajani et al., 2017]. As opposite to GAN, Generative Moment-Matching Network (GMMN) [Li et al., 2015] employs a simpler criterion to discriminate the generated and real samples. Specifically, GMMN adopts similar generator with GAN to map a simple distribution to a complicated sample distribution. Unlike GAN using a discriminator to distinguish the generated and real samples, GMMN employs maximum mean discrepancy (MMD) as the metric to measure the distance between the generated and real sample distributions. This objective is easy to optimize as long as the kernel function is smooth [Ren et al., 2016].

However, both GAN and GMMN lack the inference capability. They can only generate samples, failing to infer the latent information from observed samples. As we know, the latent code is important for model explanation and downstream tasks. Thus, some researchers propose to endow generative models with the inference capability. For instance, Adversarially Learned Inference (ALI) [Dumoulin et al., 2016] and Bidirectional Generative Adversarial Networks (BiGAN) [Zhang et al., 2016] are two representatives that can learn generative and inference models simultaneously. The basic idea is to incorporate an inference network to infer the latent code from observed samples and learn a discriminator to discriminate two joint distributions. However, these models are based on the GAN framework. They also need to solve a min-max problem, which is difficult to optimize. Additionally, these techniques cannot be applied to GMMN framework directly, since these two frameworks employ different learning schemas. Thus, it is important and not trivial to endow GMMN framework with the inference ability.

Furthermore, existing works, such as ALI and BiGAN, only focus on matching two joint distributions, ignoring the structure of the latent code distribution. As a good inference model, it should guarantee the inferred latent code distribution to be consistent with the sample distribution. Thus, it is important and necessary to preserve this consistency in the inference model.

3 Background

In this section, we will give some preliminary knowledge about kernel embedding of distributions and how to measure the distance between two distributions based on the kernel embedding.

3.1 Kernel Embedding of Distributions

To measure the similarity of two distributions, a widely used approach is to compute their distance based on their embeddings in a reproducing kernel Hilbert space (RKHS). Here, RKHS is a Hilbert function space in which each probability distribution is an element. As a Hilbert function space, RKHS

denoted by H is associated with the inner product ⟨·, ·⟩H as well as a kernel k(x, x′ ), satisfying f(x) = ⟨f(·), k(x, ·)⟩H for all elements f in this function space. Here, k(x, ·) denotes a feature map ϕ(x) implicitly such that k(x, x′ ) = ⟨ϕ(x), ϕ(x′ )⟩H.

The kernel embedding of a distribution P is defined to take expectation on its feature map [Sriperumbudur et al., 2010; Gretton et al., 2012]:

where x ∈ X ∼ P. Importantly, the kernel embedding µP is guaranteed to be an element of RKHS if EX[k(x, x′ )] ≤ ∞. More importantly, if the kernel is characteristic, all of the statistical features of a distribution will be preserved by the kernel embedding. Thus, kernel embedding is a good representation of the probability distribution.

3.2 MaximumMean Discrepancy

MaximumMean Discrepancy (MMD) is a basic tool to conduct two-sample test [Gretton et al., 2012]. Specifically, given two distributions P and Q, two datasets X = {xi}ni=1 and Y = {yi}ni=1 drawn from P and Q respectively. MMD is to test whether P = Q based on observed samples X and Y . Formally, MMD is defined as follows:

Definition 1. (Maximum Mean Discrepancy (MMD)) [Gretton et al., 2012] Given the unit ball F ⊂ H, and samples X and Y drawn from distributions P and Q respectively, the MMD is

According to [Gretton et al., 2012], under mild conditions, MMD can be reformulated as the difference between their kernel embedding:

In practical applications, we can use the empirical kernel embedding to approximate it as follows:

where {xi}ni=1 are drawn i.i.d from P and {yi}mi=1 are drawn i.i.d from Q. After obtaining the value of MMD, we can conclude the two-sample test result according to the following Theorem.

Theorem1. [Gretton et al., 2012] For a characteristic kernel F, MMD[F, P, Q] = 0 ⇐⇒ P = Q.

Thus, MMD is a reasonable metric to measure the similarity of two distributions.

4 Method

In this section, we will present our proposed model, including a joint generative moment-matching network, and a multimodal regularization to guarantee the consistency between the sample distribution and the latent code distribution.

4.1 Joint Generative Moment-Matching Network

At first, we introduce Joint Maximum Mean Discrepancy (JMMD), which will be used for computing the distance between two joint distributions.

Joint Maximum Mean Discrepancy

Similar with kernel embedding of ordinary distributions, we can define the kernel embedding of joint distributions. Formally, for a joint distribution PX,Y , its kernel embedding is defined as follows:

where ⊗ denotes the tensor product, ϕ(x)⊗ψ(y) denotes the feature map in the tensor product Hilbert space [Song et al., 2009; 2010; Long et al., 2016]. Similarly, it has the following property:

Based on the kernel embedding of joint distributions, the Joint Maximum Mean Discrepancy (JMMD) can be defined as follows.

Definition 2. (Joint Maximum Mean Discrepancy (JMMD)) Given samples (X, Y ) ∼ PX,Y and (X′ , Y ′ ) ∼ QX′ ,Y ′ , and the unit ball F ⊂ H, G ⊂ H in a RKHS, then the JMMD is

Therefore, JMMD can be represented by the kernel embedding as follows:

For real-world applications, we can compute the empirical JMMD as follows:

Afterwards, we can use ˆ JMMD[F, G, P,Q] to compute the distance between two joint distributions empirically.

Joint GMMN

Now we are ready to present the Joint Generative MomentMatching Network (JGMMN) model and how to train it by the JMMD metric.

As shown in Figure 1, JGMMN includes two networks: the generation network and the inference network. Generally, the generation network works as a generative model to generate samples. The inference network defines an inference model to infer latent codes from observed samples. According to [Dumoulin et al., 2016], this model can be formulated as a joint distribution matching problem. After matched, the desired distribution can be learned.

Specifically, the generation network works similarly with the original GMMN [Li et al., 2015] model. The basic idea of this model is that there is a map gθ between a regularly simple distribution PZ and a complicated sample distribution QX such that g(Z) ∼ QX where Z ∼ PZ , and such a map can be represented by a deep neural network due to its flexibility and powerful capability on fitting nonlinear functions. Based on this idea, the generation network starts from a stochastic layer where each unit independently follows a uniform distribution as follows [Li et al., 2015]:

where U(z) = I(-1 ≤ z ≤ 1) is an uniform distribution in [-1, 1]. Afterwards, several non-linear hidden layers follow this stochastic layer, mapping z to a point ˆx in the sample space as follows:

where ˆx ∈ ˆX is the generated sample. This process defines the conditional distribution PX|Z. Here, gθ denotes a multilayer deep neural network, which is parameterized by θ. Due to the powerful capability of the deep neural network in fitting nonlinear functions, this process can characterize PX|Z well. As a result, we can obtain the joint distribution with respect to the prior Z and the generated ˆX as follows:

However, the process defined in Eq. (11) can only generate samples. In many tasks, we need to infer the latent code from observed samples X for interpretation and downstream tasks. Hence, we propose the inference network as shown in Figure 1. Specifically, the inference network learns a transformation hβ which maps observed samples X ∼ QX to latent codes ˆ Z ∼ QZ|X. Formally,

Then, we will have the joint distribution from the inference network as follows:

The proposed JGMMN model is to match these two joint distributions defined in Eq. (12) and Eq. (14). If they are matched, the corresponding marginal distributions and conditional distributions also can be guaranteed to match [Dumoulin et al., 2016]. Especially, the conditional distribution QZ|X defined by the inference network will match the desired posterior distribution PZ|X.

To match these two joint distributions, we train the generation and inference network by optimizing the JMMD objective as follows:

where {(xi, ˆ zi)}ni=1 ∼ QX,Z is from the inference network, {(ˆxi, zi)}mi=1 ∼ PX,Z is from the generation network. By optimizing this objective function, we can learn model parameters for both generation network and inference network simultaneously. Then, our model will have both generative and inference capabilities.

4.2 Structural Latent Code

Although matching two joint distributions can guarantee the match between the conditional distribution QZ|X and the desired posterior distribution PZ|X, it cannot guarantee the learned QZ|X consistent with the sample distribution QX. Specifically, the inferred latent distribution QZ|X should have similar structures with the sample distribution QX. More specifically, if the sample distribution QX is a multi-modal distribution, QZ|X should also be a multi-modal distribution such that the inferred latent code is a good representation of observed samples. Additionally, the multi-modal property is very common in real-world applications. Discovering this property is helpful for many downstream tasks, such as clustering on the latent code. Thus, it is important to discover this multi-modal structure and preserve it in the latent space.

To address this problem, we propose an unsupervised multi-modal regularization to guarantee the consistency between QX and QZ|X. To this end, we need to discover the multi-modal distribution in both QX and QZ|X first, and then align them. Formally, we define two auxiliary distributions p and q, where p denotes the modality assignment distribution of the latent code and q denotes that of observed samples. Specifically, assume that there are c modalities, then pij denotes the probability of the i-th latent code belonging to the j-th modality, and qij has the same meaning for the i-th observed sample. By matching these two auxiliary distributions p and q, we can guarantee the consistency between QZ|X and QX implicitly. To this end, we employ KL-divergence to match them as follows:

Now, the naturally following question is how to represent p and q seamlessly with previous networks' architectures.

For the modality assignment distribution p of the latent code, we propose to compute it as follows:

where zi is the latent code of the i-th samples, wj is the model parameter associated with the j-th modality which can be learned jointly with network parameters. Actually, this is a softmax operation, which predicts the probability of the i-th latent code belonging to the j-th modality. Interestingly, this operation is to add a softmax layer on the top of the inference network, just as shown in Figure 1, which can be learned jointly with the other two networks' parameters.

For the modality assignment distribution q of observed samples, it is usually not available in real-world applications. Thus, how to find q is challenging. In this paper, we propose a novel strategy to compute q dynamically. Specifically, since qij denotes the probability of the i-th sample belonging to the j-th modality, it can be initialized by K-means. In this way, each cluster corresponds to one modality, and the clustering assignment can be viewed as the modality assignment. Note that we didn't performK-means on the original samples. Instead, we pretrain the inference network in the layer-by-layer way, obtaining new representations of observed samples, then we perform K-means on the new representation to get the desired initialization. Afterward, we refine q iteratively since such an initialized distribution q is not perfect. Specifically, after obtaining network parameters and p, we fix them and optimize Eq. (16) to update q, which is easy to solve by setting the gradient to zero. Then, we fix q to update network parameters and p.

Finally, the objective function of our model is:

where θ and β are model parameters of the generation and inference network respectively, w is the model parameter of the softmax layer. This model can be optimized easily by stochastic gradient descent method.

5 Experiments

5.1 Synthetic Data

At first, we conduct two experiments on the synthetic dataset. Specifically, we construct two synthetic datasets. The first dataset shown in Figure 2(a) is drawn from a 2D Gaussian mixture distribution Q1X which has 4 components. The second dataset shown in Figure 2(e) is more complicated than the first one. It is drawn from a 2D Gaussian mixture distribution Q2X which has 9 components. For both of them, we draw 2,000 samples.

Both the generation and inference network of the first synthetic dataset is a 3-layer MLP: [100, 100, 2]. Similarly, that of the second synthetic dataset is [100, 300, 2]. The activation function employed is ReLU [Nair and Hinton, 2010]. Note that the last layer employs the linear activation. The size of mini-batch is set as 500. The kernel employed in JMMD is the Gaussian kernel. Here, we use a mixture of several Gaussian kernels, that is k(xi, xj) = ∑Mm=1 km(xi, xj) where different kernels have different bandwidth parameters. In this paper, the bandwidth employed is {2.0, 5.0, 10.0, 20.0, 40.0, 80.0}.

The result is shown in Figure 2. From Figure 2(b) and 2(f), we can find our proposed method can generate similar samples with the real ones, which means our approach has learned a good generation network. Figure 2(d) and 2(h) show that the inferred latent code follows a multi-modal distribution as the sample distribution. Thus, the inference network learned from our model can capture the intrinsic latent structure in the dataset. In Figure 2(c), the latent code is learned without regularization, that is Eq. (15). While, in Figure 2(d), the latent code is learned from our proposed method with multimodal regularization, that is Eq. (18). Comparing these two figures, we can find that the latent code learned with regularization has a better structure than that without regularization. Specifically, the margin among different modalities is larger and each modality is more compact, which verifies the effectiveness of our proposed regularization technique. Similar observations can be found from Figure 2(g) and 2(h). In summary, all these observations have confirmed the effectiveness of our proposed JGMMN.

5.2 Real-World Data

To further show the performance of our proposed model, we conduct experiments on five real-world datasets, which includes 3 image datasets: MNIST [LeCun et al., 1998], USPS [Cai et al., 2011] , ExtendYaleB (EYB), and 2 text datatsets: Reuters-10K [Xie et al., 2016], 20News 1 . We summarize the statistics of these datasets in Table 2. The network architectures (from the 2nd layer) for these datasets are summarized in Table 3. Both the generation and the inference network consist of a 4-layer MLP. In our experiments, the generation network employs ReLU in all layers except the last layer. Sigmoid activation function is used in the last layer. The inference network also utilizes ReLU in all layers other than the last layer. The linear activation function is used in its last layer. The number of modalities c is set as the number of classes. The used kernels are same as those of the synthetic dataset.

The Generative Network: In Figure 3, we show the real and generated samples of three image datasets. Specifically, the fourth column in Figure 3 is the generated samples of our proposed JGMMN. The second and third columns are those of ALI [Dumoulin et al., 2016] and GMMN [Li et al., 2015] respectively. Although the generated samples of GMMN and JGMMN is a little obscure than those of ALI due to the intrinsic property of kernels [Li et al., 2017], most generated samples of JGMMN are very similar with real samples and those of baseline methods, which confirms the correctness of our JGMMN.

1 http://qwone.com/˜jason/20Newsgroups/

The Inference Network: To show the performance of the inference network of JGMMN, we perform clustering on the obtained latent code. Here, we directly use the output of the softmax layer defined in Eq. (17) as the clustering result. Then, we compare our method with four baseline methods: K-means, AE+K-means which performs K-means on the learned feature from autoencoder, VAE+K-means which performs K-means on the learned feature from variational autoencoder and DEC [Xie et al., 2016]. Note that, to make a fair comparison, we didn't compare it with those CNN-based deep clustering methods. Here, we only focus on the DNN structure and ensure these DNN-based methods have consistent network architecture. Here, we use Clustering Accuracy (ACC) and Normalized Mutual Information (NMI) to evaluate the performance the clustering performance. From Table 1, we can find our proposed JGMMN has achieved the best result for most cases. In conclusion, the inference network of our proposed JGMMN can learn meaningful latent code from samples.

Additionally, to verify the effect of the multi-modal regularization, we compare our method with JGMMN-NR which corresponds to Eq.(15). The clustering result is shown in Figure 4. Here, we only show the result of MNIST dataset. The other datasets have similar results. We can find that JGMMNNR has very bad clustering performance, which means that its inference network fails to learn discriminative latent representations. Thus, the proposed modal regularization is very critical to learn a good inference network.

6 Conclusion

In this paper, we propose a novel Joint Generative MomentMatching Network, which can simultaneously learn a generative network and an inference one. To learn them, we employ the Joint Maximum Mean Discrepancy to match two joint distributions. Furthermore, we propose a novel multi-modal regularization to enforce the latent distribution to be consistent with the sample distribution. Extensive experiments have verified the effectiveness and correctness of our proposed JGMMN model.

Acknowledgements

This work was partially supported by the following grants: NSF-IIS 1302675, NSF-IIS 1344152, NSF-DBI 1356628, NSF-IIS 1619308, NSF-IIS 1633753, NIH R01 AG049371.