Discriminative Sample Generation for Deep Imbalanced Learning

1 Introduction

Learning and classification of data with imbalanced class distributions is a significant challenge for the machine learning community. On one hand, minority class instances often represent the objects of interests in applications [He and Garcia, 2008; Sun et al., 2007; Chen and Shyu, 2011], and require significant attention by the underlying learning algorithms. On the other hand, class distributions have a significant impact to most machine learning methods, and with a disadvantage in populations, minority class instances are essentially more vulnerable being incorrectly classified by learning algorithms [Krawczyk, 2016]. Accordingly, many methods exist to (1) modify sample distributions using random underor over-sampling [Wang et al., 2015; Chawla et al., 2002; Barua et al., 2012], or (2) modify costs associated to different classes [Yan et al., 2017], such that the learning and classification can emphasize on minority class instances. For most researches in the field, they are carried out in the conventional machine learning context and work on the original feature space for learning and classification.

∗Contacting Author

Recently, deep learning has emerged with superb performance gain, thanks to the powerful feature learning. However, most existing methods are mainly designed under assumptions that training data are relatively balanced and do not take the class distributions into consideration for feature learning. A couple of works have recently addressed deep learning for imbalanced data [Wu et al., 2017; Yan et al., 2015], whereas these methods are all based on sampling approaches working on the original instance space to learn new features. These methods do not have a solution to differentiate class boundaries, so the feature learning cannot be aligned to the class distributions. This essentially deteriorates deep learning performance for imbalanced data.

While learning from imbalanced data has been severely challenged by the imbalance and scarcity of minority class samples, latent space generative models in deep learning, for example, variational autoencoders (VAEs) and generative adversarial networks (GANs) have already provided effective ways to learn mapping from a latent encoding space to a data space [Kingma and Welling, 2014; Goodfellow et al., 2014]. In other words, VAEs or GANs can generate new instances resemble to the original data, and this raises an open question on whether synthetic minority class samples can be generated to alleviate imbalanced learning challenges.

Indeed, VAEs and GANs have shown to generate samples close to original data, but such samples are normally not directly usable to help re-train the model. On one hand, VAEs suffer from the effect of L2 loss and thus generate blurry images, because L2 loss is based on the assumption that the data follow a single Gaussian distribution. When samples have multi-modal distributions, VAE cannot generate samples with sharp edges and fine details. Besides, the squared error is an element-wise measure which lacks higher-level and sufficiently invariant discriminative information to guide the training process. GANs, on the other hand, learn a loss that tries to classify if the output sample is real or fake, while simultaneously training a generative model to minimize this loss [Goodfellow et al., 2014]. GANs suffer from training instability and the quality of generated samples are largely not as good as original data and may cause over-fitting problem. Accordingly, some researches suggested hybrid VAE- GAN model by combining a variational autoencoder with a generative adversarial network in which learned feature representations in the GAN discriminator can be used as the basis for the VAE reconstruction objective [Larsen et al., 2016; Dosovitskiy and Brox, 2016; Yu et al., 2017]. In addition, to allow users to have control on the sample distributions, conditional VAE/GAN (CVAE/CGAN) was developed [Mirza and Osindero, 2014; Isola et al., 2017; Sohn et al., 2015; van den Oord et al., 2016; Bao et al., 2017], so users can generate samples close to a user-specified category.

Although CVAE/CGAN can be conditioned to one additional variable c expressing the labels. They are not suitable for imbalanced learning as they are unable to differentiate class boundary in latent space. Take Fig.1 (a) as an example, Z is the latent representation and P(Z) is the latent distribution where positive and negative samples are mixed together and positive samples can only be distinguished from negative ones when specific c = positive is given and thus P (Z|c = positive) ∼ N (0, I).

The above observations motivate us to model latent representation as being drawn from two simple distributions with opposite means, which can be adjusted through the training of adversarial networks. As a result, we can explicitly depict the class boundary in latent space to help improve the discriminator's performance. By clearly modeling class boundary to generate new minority class samples, we can directly learn a classifier by learning discriminative latent features and generating high-quality synthetic minority class samples to alleviate class imbalance for deep imbalanced learning.

The main contribution of the paper is threefold:

Discriminative generative model. We impose a prior over the latent two-component mixture distribution to explicitly differentiate different classes in the latent space.

Adversarial networks for class boundary alignment. We unify jointly learn and align class boundary in the latent space to generate high-quality synthetic samples to help improve the discriminator.

Data augmentation for deep imbalanced learning. Different from the existing sampling-based approaches for imbalanced learning, we propose to generate and include synthetic samples close to the class boundary (support samples) to alleviate the class imbalance. This allows discriminative and generative models to be seamlessly integrated for effective deep imbalanced learning.

2 ProblemDefinition and Preliminary

2.1 ProblemDefinition

For deep imbalanced learning, we are given a training set with imbalanced binary class labels. The class presented with very few samples but associated with higher identification importance is referred to as the minority (or positive) class, while the other one is taken as the majority (or negative) class. Mathematically, given a training dataset D = {< xi, yi > |xi ∈ Rn , yi ∈ {-1, 1}}, Xpos = {xi|yi = 1} and Xneg = {xi|yi = -1}, we aim to learn a deep discriminative model (i.e. a classifier) fromD with the best performance in classifying both minority and majority class test samples.

2.2 Variational Autoencoder

A traditional VAE consists of two networks that encode a data sample x to a latent representation z and decode the latent representation back to data space, respectively:

The VAE regularizes the encoder by imposing a prior over the latent distribution P(z). Typically z ∼ N(0, I) is chosen, where I is the identity matrix. The VAE loss is negative of the sum of the expected log likelihood (the reconstruction error) and a prior regularization term:

where DKL is the Kullback-Leibler divergence.

2.3 Generative Adversarial Network

A GAN consists of two networks: a generator Gen(z) maps latent z to data space while a discriminator network assigns probability y = Dis(x) ∈ [0, 1] that x is an actual training sample and probability 1 - y that x is generated by proposed model through x = Gen(z) with z ∼ P (z). The GAN objective is to find the binary classifier that gives the best possible discrimination between true and generated data and simultaneously encouraging Gen to fit the true data distribution. We thus aim to maximize the binary cross entropy:

with respect to Dis/Gen with x being a training sample and z ∼ P (z).

3 DVAAN for Deep Imbalanced Learning

Fig. 2 shows the proposed Discriminative Variational Autoencoding Adversarial Network (DVAAN), which includes a discriminative variational autoencoding (DVAE) module and a generative adversarial network (GAN) module.

3.1 Discriminative Variational Autoencoding

For existing VAE-based methods, they assume that the latent representation follows a simple normal distribution N(0, I) without considering the class variety. This is, unfortunately, not the case for real-world data. For example, statistics show that men have an average taller figure than women. If we take height as a feature, men and women will each fall into different intervals and the overall distribution of human is known not following any unimodal distribution [Schilling et al., 2002]. Besides, as the imbalance degree may cause the unreliability of uncovering regularities inherent in a small class, the assumption of one simple distribution over all classes may be biased to the majority.

Alternatively, we regularize the encoder by imposing a prior over the latent two-component mixture distribution P(z|y). It is a conditional probability distribution (CPD). It means for each possible value of y, we would have a P(z).

µ is a chosen parameter. In DVAE, we assume that the latent representations follow a two-component mixture distribution. By forcing discriminative distributions to different classes, the potential discriminative regularities inherent in the minority class can be retained to a greater extent without being lost in the learning preference on the majority class. Besides, it is obvious that for different classes, the distribution of the data should be different. Therefore it makes more sense to choose different distributions for different classes.

As the proposed Discriminative VAE is a supervised generative model, the class information (label) is known for training process. Therefore, we have,

The loss function of the Discriminative VAE is the expected log likelihood with a regularizer:

with

The loss function is similar to Conditional VAE (CVAE) [Sohn et al., 2015], which uses class information y as part of the inputs for both encoder and decoder to model latent variables and data conditioned to some random variables. However, rather than considering y as extra input, DVAE uses y to decide zpos ∼ N(µ, I) or zneg ∼ N(-µ, I), which are used as the prior to impose for the latent distribution P(z). It means that zpos and zneg are in the same latent space while the latent representations of CVAE for different classes are in different spaces. By adjusting µ, the boundary of positive and negative classes are changing accordingly.

3.2 DVAAN Framework

The appealing properties of GAN are that its discriminator network implicitly learns a rich top-down similarity metric for classification tasks and this discriminator can be considered as a classifier. We thus propose to transfer the properties of data learned by the discriminator into (1) a more abstract reconstruction error for the DVAE, and (2) a classifier for distinguishing positive samples from negative ones. The end result will be a method that combines the advantage of GAN as a high quality generative & classification model and VAE as a method that produces an encoder of data into the latent space z with minimal information loss.

Specifically, we use the decoder network of the DVAE as the generator network of the GAN, but instead of training a separate discriminator network, we combine the decoder and discriminator into a single network. So our inputs for GAN

Algorithm 1 Training the DVAAN model

input: Imbalanced training dataset D output: θDis µ, θEnc, θDec, θDis ← initialize network parameters; D+ ← {}

repeat

// Update parameters according to gradients

until Convergence

is ˜ x = Gen(z) = Dec(z) where z ∈ {zpos, zneg}. The task for the GAN discriminator is twofold: (1) to learn a top-down similarity metric to discriminate positive from negative generated/real samples. We thus aim to maximize/minimize the binary cross entropy: Ladv GAN , which is the adversarial loss that forces the discriminator to label a sample as positive or negative; (2) to minimize the feature-wise reconstruction loss, Lf ea GAN , by evaluating the difference between original and reconstruction in the space of the hidden layers of the discriminator.

We train our combined model with the triple criterion:

Class Boundary Alignment

In DVAE, we introduce a latent two-component mixture distribution P(z|y) to depict the latent representation in hidden space. As µ is a predefined parameter, if µ is too large, the generator network is hard to learn identical latent representations zpos and zneg because of inseparability of original data representation. While if µ is too small, it is hard for discriminator network to identify the positive samples from negative ones as the inseparability of latent representation zpos and zneg. In this section, we will propose an update strategy for µ based on the adversarial loss. It is a modification of the adversarial loss that forces the class boundary of latent representation to better minimize both top-down reconstruction loss (Lfea GAN ) and final classification loss (Ladv GAN ).

tanh() is the hyperbolic tangent function and we use a parameter γ to weight the ability to update µ. So based on Eq. (10), we can update µ by using µk+1 = µk - ∆µ, where k is the iterations. It is obvious that if the discriminator network is hard to identify positive reconstructed samples from negative ones (Ladv GAN accounts for greater), we increase µ to let latent distributions be far away from each other (∆µ > 0) to make latent representations more discriminative; while if the decoder/generator is hard to reconstruct original samples with small reconstruction loss (Lfea GAN accounts for greater), we decrease µ to make latent representations from different classes close to each other and therefore ensuring highquality reconstruction ability.

Data Augmentation

Benefit from the strong generative ability of GAN and the control ability of the forced two-component mixture distribution of DVAE, we can generate synthetic samples close to the class boundary, and these samples can be considered similar to support vectors defined in SVM [Cortes and Vapnik, 1995].

Specifically, as we force our latent representation to follow a two-component mixture distribution by given µ, the samples generated from Z0 ∼ N(0, I) are highly likely to be the support samples. Accordingly, we first generate a latent representation z from Z0 ∼ N(0, I) and calculate the distance between z to µ and -µ respectively. If z is close to µ we set a pseudo label to z as y = 1 and y = -1 otherwise. Then we generate the support samples ˜x = Dec(z), and we will get another pseudo label y′ = Dis(˜x). If y′ = y, we have the confidence to consider ˜x as an additional training samples.

Algorithm 1 lists the training procedure of DVAAN.

4 Experiments

4.1 Benchmark Data

MNIST. is a handwritten digit database commonly used for deep learning. It has a training set of 60,000 examples, and a test set of 10,000 samples 1 .

Fashion-MNIST. is a dataset of Zalando's article images-consisting of a training set of 60,000 examples and a test set of 10,000 samples. Each example is a 28x28 grayscale image, associated with a label from 10 classes 2 .

CIFAR-10. is one of themost widely used datasets for deep learning. It contains 60,000 32x32 color images in 10 different classes, representing airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks 3 .

TIS. is used to predict the Translation Initiation Sites (TIS) at which the translation from a messenger RNA to a protein sequence was initiated. The dataset has two classes and 927 features 4 . The detailed process, including a biological explanation, is detailed in [Liu and Wong, 2003].

Because MNIST, Fashin-MNIST and CIFAR-10 are multiclass datasets, we choose two similar classes from each dataset. Digit 5 (positive) and Digit 6 (negative) are chosen from MNIST, Sandal and Sneaker are chosen from FashionMNIST, and cat (positive) and dog (negative) are chosen from CIFAR-10. For all four datasets, we randomly sample a percentage of positive training samples to create different imbalance levels. For example, an imbalance ratio 1:10 is created by selection 10% of positive samples and keep 100% negative samples (except for Table 2 which we use one-against-all meaning we keep all positive samples and all samples in other classes are used as negative).

1 http://yann.lecun.com/exdb/mnist/

2 https://github.com/zalandoresearch/fashion-mnist

3 https://www.cs.toronto.edu/∼kriz/cifar.html

4 https://www.comp.nus.edu.sg/∼wongls/courses/cs2220/2008/ TIS-data.html

4.2 Baseline Methods & Settings

For all baselines, the encoder network consists of four convolution layers followed by two fully-connected layers. The convolution layers have 64, 92, 128 and 256 kernels with the filter size of 5×5, 5×5, 3×3 and 3×3. The decoder network (Generator network) consists of 2 fully-connected layers, followed by 4 deconv layers with 2-by-2 unsampling and corresponding architectures to the encoder network (both kernels and filter size). For the Discriminator network, we use the same discriminator network as the VAE+GAN [Larsen et al., 2016]. The batch normalization layer [Ioffe and Szegedy, 2015] is also applied after each convolution layer. The model is implemented using Tensorflow toolbox.

VAE-GAN. is an updated version of VAE-GAN [Larsen et al., 2016] for binary classification task. The difference is the input of GAN's discriminator is the reconstructed positive and negative samples and the aim of GAN's discriminator is to distinguish positive samples from negative ones.

VAE-GANb. uses over-sampling techniques for mini-batch selection to balance the class distribution. This serves as a baseline of traditional sampling methods for deep learning from imbalanced data.

AC-GAN. adds a classifier layer to the discriminator and a conditioning vector to the generator [Odena et al., 2016]. The discriminator is then trained to minimize classification error in addition to the traditional GAN objective of real vs. fake. This allows labels to be leveraged and provides additional information to the GAN. In addition, it allows the generator to be conditioned to produce certain classes of samples.

CVAE-GAN. combines a variational auto-encoder with a generative adversarial network for synthesizing images in fine-grained categories [Bao et al., 2017]. This approach models a sample as a composition of the label and latent attributes in a probabilistic model.

DVAAN, DVAANµ andDVAANa. DVAAN is the full version of our proposed algorithm. To validate the effectiveness of the class differentiation in the latent space, we fix µ = 3 and denote this method by DVAANµ. Meanwhile, to validate the data augmentation, we drop the data augmentation component from DVAAN (so no synthetic data are added to the training set), and denote this variant by DVAANa.

4.3 Classification Performance Comparison

Table 1 reports the AUC scores of different methods. The results show that VAE-GAN has the worst performances because no treatment was used to tackle imbalanced data in learning. Meanwhile, the results from VAE-GANb show that over-sampling is effective to help improve the learning performance on imbalanced data, mainly because the balanced sample distributions of mini-batch selection in VAE-GANb helps avoid the preference of optimizing reconstruction and adversarial loss of the samples from the majority class.

Overall, AC-GAN and CVAE-GAN achieve a better performance comparing to VAE-GAN as they take label information into consideration to train a better generative model and therefore high-quality generated samples could help improve the performance of the discriminator.

In Fig. 4, we report the change of the latent two-component distribution during the training. For MNIST, the converged µ value is very small which means that the latent distribution over two classes tend to follow a unimodal distribution. This explains why VAE-GANµ, which forces a simple unimodal distribution on latent representations, has a relatively good performance on MNIST as well. On the contrary, TIS has a relatively large converged µ value meaning that the latent distributions of positive and negative classes are different, so patterns between classes are overlapping at certain levels in feature space. As a result, discriminative rules are hard to induce so the model tend to increase µ to make patterns more separated for features. The flexibility of µ allows our model to better handle separability problem of imbalanced learning.

Comparing DVAANa and DVAAN in Table 1, it is clear that data augmentation indeed helps improve the classification accuracy as synthetic samples generated close to the class boundary help discriminators learn better discriminative model. Similar to the core idea of SVM, the training aim of our discriminator is to find a mapping so that the support samples (vectors) of the separate categories are divided by a clear gap that is as wide as possible.

In Table. 2, we change the setting to one-against-all. The results show that our algorithm still remain good performance even if negative classes have complex distributions.

Figure. 5 reports the performance on TIS dataset by using different imbalance ratios. The results show that our model have better performance, especially for high imbalance ratios. This confirms that data augmentation, using synthetic samples to help characterize class boundary, is particularly useful when positive samples are sparse. Different from traditional sampling methods, DVAAN generates samples resemble to the training data of the same class and close to the class boundary and uses them to alleviate class imbalance.

4.4 Class Boundary Learning Results

In Fig. 6, we use DVAAN to generate samples from different latent distributions on MNIST dataset, so we can visually examine the quality of generated samples. Specifically, the first row and last row show the samples generated from two-component latent distribution. These samples can be clearly recognized with their belonging classes meaning that the learned latent representations indeed follow our proposed two-component latent distribution. The middle two rows are generated from a simple distribution with µ = 0. These samples are more likely to be the ones close to the class boundary in latent space. For example, the second row of Fig. 6 is hardly recognized to belong to positive (digit 5) or negative (digit 6) class. It means these samples tend to share similar features/feature values. Adding these samples to the training data can help discriminator better learn the separating capability based on the shared similar features/feature values.

4.5 Reconstruction Error Comparison

Table 3 reports the reconstruction errors of VAE-based methods. The results show that CVAE-GAN has the least average MSE on MNIST, Fashion-MNIST, and CIFAR-10, because CVAE-GAN take label information as the latent condition for both encoder and decoder of the model. Even though the latent representations for positive and negative classes are in same latent space, they are somehow independent as the forced simple distribution is conditional to different classes and therefore CVAE enjoys a more flexible learning process for each single class. It is worth noting that DVAAN focuses on finding features best separating positive and negative samples rather than finding features with least information loss.

5 Conclusion

In this paper, we proposed a Discriminative Variational Autoencoding Adversarial Network (DVAAN) for deep imbalanced learning. We argued existing deep learning methods for imbalanced data cannot differentiate class boundaries, so their feature learning cannot be aligned to the class distributions to differentiate minority class from the majority class. In comparison, DVAAN combines a discriminative generative model and an adversarial network to explicitly learn class boundary in the latent representation space. As a result, it can control the generation of synthetic samples to alleviate class imbalance for deep imbalanced learning.

Acknowledgements

This research is sponsored in part by the US National Science Foundation through Grants No. IIS-1763452 and CNS1828181.