Stacked Similarity-Aware Autoencoders

1 Introduction

In the recent years, deep neural networks (DNN) have been developed and used successfully to learn rich and powerful visual representations [Krizhevsky et al., 2012]. However, applying this approach usually brings about a huge burden as it demands millions of labeled examples. A natural way to address this problem would be to combine the powerful DNN model and the idea of unsupervised learning so that we can extract useful features without any annotations. Then with limited label information, the learned representations will be more suitable as input to a supervised machine than the raw input. Among massive unsupervised learning models, the autoencoder could be stacked to build a deep structure easily and then utilized to generate useful features naturally [Hinton and Salakhutdinov, 2006]. Not surprisingly, autoencoder based models have been shown to achieve state-of-the-art performance in a number of challenging problems ranging from computer vision [Wang et al., 2016] and audio processing [Plchot et al., 2016] to natural language processing [Wang et al., 2015].

∗ corresponding author

The conventional autoencoder is based on an encoderdecoder paradigm. Specifically, the encoder first transforms the input into a latent representation, and then the decoder is trained to reconstruct the initial input from this representation as far as possible. This architecture forces the latent representation to capture the most important information of the training data. Unfortunately, if the encoder and decoder are allowed too much capacity, the autoencoder may just learn to perform the identity function without extracting useful representations [Goodfellow et al., 2016]. Thus, there are many studies presenting advanced regularization techniques which have led to a series of variants including denoising [Vincent et al., 2010], contractive [Rifai et al., 2011b], k-sparse [Makhzani and Frey, 2014] and winnertake-all [Makhzani and Frey, 2015] autoencoders.

However, the conventional autoencoder and most of its variants only consider the one-to-one reconstruction, which ignores the geometric structure of the data and may lead to overfitting. Generally speaking, the samples of the collected dataset can be grouped into several categories with the similar semantic. This intrinsic characteristic requires that their ideal discriminative representations should form several clusters in the high-dimensional feature space. For example, the MNIST database of handwritten digits has ten categories. Here we train a Convolutional Neural Networks (CNN) model [Lecun et al., 1998] on the MNIST dataset and use t-SNE [Maaten and Hinton, 2008] to visualize different layer features of the samples. From Figure 1, it can be observed that for the original data space, the samples spread throughout the whole space. Interestingly, the samples mapped to the feature space from low-level to high-level, gather together and come into several groups gradually. It is obvious that these high-level representations are very useful for tasks like classification or retrieval. This observation inspires us to introduce a novel regularization technique to guide the unsupervised feature learning.

In this work, we develop a novel autoencoder based framework called Stacked Similarity-Aware Autoencoders (SSAAE). In order to capture the geometric structure in the data, we exploit similarity constraint as a generic prior on the hidden codes of the autoencoder. Specifically, we first obtain the pseudo class label of each sample by clustering the input features. Then for those samples belonging to the same category, their hidden codes of the autoencoder will be constrained to be close to each other in the high-dimensional space. Here we employ an extension of the center loss [Wen et al., 2016] to implement this similarity constraint. As a result, the learned feature representations not only can reconstruct the original data, but also preserve the geometric structure in the inputs. Even though the clustering results maybe not perfect. For example, two samples have a different real classes but same pseudo class or the same real class but different pseudo classes. However, we utilize the reliable hierarchical clustering method and only several outliers will not mislead the feature learning. Moreover, a stacked framework is utilized to improve the representation capacity. With respect to the deeper stacked autoencoder, the amount of the classes used for clustering will be set less to learn more compact highlevel representations. To the best of our knowledge, such autoencoder based deep learning scheme has not been discussed before. We conduct extensive experiments on several benchmark datasets including MNIST and COIL100. The experimental results demonstrate the superior performance of the proposed algorithm compared with other autoencoder based approaches.

The rest of this paper is organized as follows. In Section 2, we present a brief review of the related works. Then we propose the Stacked Similarity-Aware Autoencoders in Section 3. The experimental results on two real-world datasets are presented in Section 4. Finally, we provide some concluding remarks in Section 5.

2 RelatedWork

In this section, we briefly review the recent work on the basic autoencoder and a series of its variants.

The basic autoencoder [Rumelhart et al., 1988] was introduced as a technique for dimensionality reduction. It is a neural network that takes a vector input x, maps it into a hidden representation z using an encoder which typically has this form:

where f is a non-linear activation function, We the encoding matrix and be a vector of bias parameters. The hidden representation z, is then mapped back into the space of x, using a decoder of this form:

where g is also a non-linear activation function, Wd is the decoding matrix and bd a vector of bias parameters. The goal of the autoencoder is to minimize the reconstruction error, which is represented by a distance between x and ˜x. The most common type of distance is the mean squared error:

The code z typically has less dimensions than x, which forces the autoencoder to learn the most important information of the training data. If the code z has as many components as x, then no compression is required, and the model could typically end up learning the identity function.

More recently, autoencoders have played a key role in the deep neural network based approaches [Hinton and Salakhutdinov, 2006] where autoencoders are stacked and trained bottom up in unsupervised fashion, followed by a supervised learning phase to train the top layer and fine-tune the entire architecture. Unfortunately, if the encoder and decoder are allowed too much capacity, the autoencoder can learn to perform the copying task without extracting useful feature representations [Goodfellow et al., 2016].

Rather than limiting the model capacity by keeping the code size small, there are many studies presenting advanced regularization techniques for non-convolutional autoencoders. Denoising autoencoder (DAE) [Vincent et al., 2010] is trained to have denoising ability by incorporating artificially random noise to the input data, and then encouraged the output to be as similar to the original undistorted input as possible. In addition, by imposing sparsity on the hidden units during training, an autoencoder can learn useful structures in the input data even if the model capacity is great enough to learn a trivial identity function. Sparsity may be achieved by additional terms in the loss function during training [Boureau et al., 2008], or by manually zeroing all but the few strongest hidden unit activations (referred to as a k-sparse autoencoder [Makhzani and Frey, 2014]). Furthermore, the authors proposed Winner-Take-All (WTA) autoencoders [Makhzani and Frey, 2015] which use aggressive dropout, where all the elements but the strongest of a convolutional map are zeroed out. Moreover, the proposed method exploits a lifetime sparsity by keeping the k percent largest activation of that hidden unit across the mini-batch samples and setting the rest of activations of that hidden unit to zero. This strategy forces the convolutional decoder to learn robust features.

Meanwhile, graph regularization was utilized to preserve the local structure of the original data space in [Yu et al., 2013; Zhao et al., 2016]. Furthermore, [Ding et al., 2016] developed a novel algorithm named as Deep Robust Encoder (DRE) with locality preserving low-rank dictionary. The core idea is to jointly optimize deep autoencoder and a clean lowrank dictionary, which can rule out noises and extract robust deep features in a unified framework.

However, most of these autoencoder based approaches ignore the geometric structure within the dataset and may lead to overfitting. Motivated by the recent work [Yang et al., 2016], in which the authors proposed to jointly perform unsupervised learning of deep representations and image clusters, we introduce a novel similarity constraint based regularization technique to guide the unsupervised feature learning.

3 The Proposed Approach

In this section, we first introduce the motivation of our proposed algorithm, followed by our detailed model named Similarity-Aware Autoencoder. Then we present the stacked architecture which is allowed more representation capacity.

3.1 Similarity-Aware Autoencoder

First, we provide the definition for the conventional autoencoder. For simplicity, we use xi, zi, ˜xi to represent the ith input vector, hidden vector and output vector respectively. Then the reconstruction error can be formulated as bellow:

From this formulation, we can see that the conventional autoencoder only consider the one-to-one reconstruction error, which ignores the geometric structure of the data and may lead to overfitting. In general, the collected dataset could have several latent concepts and some samples belong to the same concept. This intrinsic characteristic requires that their ideal representations should form several clusters, which can lead to satisfactory performance for tasks like classification or retrieval. In addition, according to our observation from Figure 1, the samples mapped to the high-level feature space, gather together and come into several groups. This observation inspires us to introduce a novel regularization technique to guide the unsupervised feature learning. Specifically, we hope that the ideal autoencoder will learn a mapping from the inputs to a compact Euclidean space, where the similarity relationships of the corresponding input vectors are preserved or even reinforced. With this similarity constraint, the learned feature will be more compact and discriminative. However, the first difficulty is that we don't have the similarity relationships or supervised signals here. Thus we want to use clustering algorithm to get pseudo class labels. For convenient, we employ the popular agglomerative clustering algorithm [Gowda and Krishna, 1978]. The core idea in agglomerative clustering is to merge two clusters at each step until some stopping conditions. More details can be found in [Jain et al., 1999].

After that, one object class is comprised of the input vectors clustered into the same group. Even though the clustering results maybe not perfect, which means two examples have a different real class but same pseudo class or the same real class but different pseudo classes. These two situations might cause problems. However, the hierarchical clustering method is reliable and only several outliers will not mislead the feature learning. Then, how to develop an effective similarity constraint function to improve the discriminative power of the deeply learned feature representations? Intuitively, minimizing the intra-class variations while keeping the features of different classes separable is the key. Inspired by [Wen et al., 2016], we utilize the center loss to supervise the learning. Suppose there are N classes for the samples and we use yi to denote the pseudo class label for xi. Here is the formulation for the center loss in [Wen et al., 2016].

The cyi ∈ Rd denotes the yith class center. And the centers are computed by averaging the hidden codes of the corresponding classes. The formulation effectively reduces the intra-class variations.

However, it didn't take the inter-class variations into account. We hope that the samples should be away from those cluster centers which have different pseudo class labels. To this end, we propose the improved center loss function, as formulated in below.

Here the γ is a margin to avoid the influence from those remote class centers who are far enough. And the β is a balance parameter for minimizing the intra-class variations and maximizing the inter-class variations. The gradients of Ls with respect to zi is computed as:

where 1 [∥∥∥zi - cy′ ∥∥∥2 2 ≤ γ ] is the indicator function which

takes a value of one if its argument is true and zero otherwise.

Ideally, the class center cyi should be updated as the hidden codes changed. In other words, we need to take the entire training set into account and average the hidden codes of every class in each iteration, which is inefficient even impractical. Therefore, we follow the solution in [Wen et al., 2016] to address this problem. First, instead of updating the centers with respect to the entire training set, we perform the update based on mini-batch in each iteration. Second, to avoid large perturbations caused by few mislabeled samples, we use a scalar α to control the learning rate of the centers. Then the update equation of cyi is computed as:

Furthermore, the objective function of the similarity-aware autoencoder can be described as below:

where λ is a hyper parameter which maintains the balance between these two loss. The framework of the proposed similarity-aware autoencoder is illustrated in Figure 2. For convenience, we append an embedding layer on the top of the hidden codes. And impose the similarity constraint on the embedding layer. As a result, the learned feature representations not only can reconstruct the original data, but also are close to each other for the similar samples.

3.2 Stacked Architecture

In this section, we introduce a stacked framework which is utilized to improve the representation capacity. The traditional stacked autoencoders will use a single autoencoder as a building block to form a deep architecture. More precisely, the hidden units of the current autoencoder are worked as the input to feed in the next autoencoder to obtain the stacked representations. According to our observation from Figure 1, the samples mapped to the feature space , gather together and come into several groups gradually from lowlevel to high-level. This phenomenon gives us an insight that the feature representations from the deeper layer should be more discriminative in term of the semantic. Therefore, the deeper stacked autoencoder deserves stronger similarity constraint to make the hidden codes capture high-level semantic information. And this can be controlled easily by adjusting the amount of classes used for clustering. To this end, for the deeper layers of the proposed stacked autoencoders, the amount of classes used for clustering will be set less to learn more compact high-level representations.

Algorithm 1 Stacked similarity-aware autoencoders

• 1: Input: cluster number Nc > 0 for agglomerative clustering, shrinking factor 0 < α < 1, iterative number R, stacked autoencoder number T > 0, and training examples {xn}Nn=1 ,
• 2: Output: Parameters of stacked autoencoder {Wt}Tt=1 ,
• 3: Initialize training examples' labels {yn}Nn=1 = 0
• 4: Compute the number of clusters Nc = ⌊N × α⌋
• 5: Run agglomerative clustering on {xn}Nn=1
• 6: Update training examples' labels {yn}Nn=1
• 7: for t = 1 to T do
• 8: for r = 1 to R do
• 9: Compute the stochastic gradient according to Eq. (1)
• 10: Update {Wt}
• 11: end for
• 12: Compute the number of clusters Nc = ⌊Nc ×α⌋
• 13: Compute the hidden vector {zn}Nn=1 with current learned autoencoder
• 14: Update {xn}Nn=1 = {zn}Nn=1
• 15: Run agglomerative clustering on {xn}Nn=1
• 16: Update training examples' labels {yn}Nn=1
• 17: end for
• 18: Return: {Wt}Tt=1

The whole procedure of the proposed stacked similarityaware autoencoders is summarized in Algorithm 1. Before the training of each similarity-aware autoencoder, we need to compute the similarity relationships between all samples. Here, we employ the agglomerative clustering algorithm to obtain the pseudo class labels. The cluster number varies from large to small when the hidden codes of the autoencoder need to represent higher level features. We will introduce a hyper parameter 0 < α < 1 to control the process. Suppose we have Nc clusters in previous autoencoder, then the next cluster number is multiplied by α. Through this way, the cluster number is shrinking for the deeper autoencoder, which is corresponding to our observation in Figure 1. The hyper parameter α is chosen so that the cluster number of the last autoencoder is approximate to the true class number in the original dataset. In addition, the agglomerative clustering continues to merge clusters from previous clustering results.

4 Experiments

We conduct experiments on two widely used datasets to evaluate our proposed SSA-AE for unsupervised feature learning and compare them with several state-of-the-art methods. The following describes the details of the experiments and results.

4.1 Results on COIL100

COIL100 contains 7,200 color images of 100 objects. The images of each objects were taken 5 degrees apart as the object is rotated on a turntable and each object has 72 images (Figure 3). The converted gray scale images with size 32 × 32 are used. We selected 10 images per object randomly to form the training set and the rest images are in testing set. The nearest neighbor classifier is applied and the inputs are the learned feature representations. The random split is repeated 10 times, and the average results are reported with standard deviations. All of the images are used in the unsupervised feature learning and other compared methods to learn the projection function.

Our SSA-AE model has two essential parameters: the balance parameter of similarity constraint and the stacked autoencoders number. First, we focus on the parameter λ in Eq. (1) which represents the importance of similarity constraint and fix other hyper-parameters. We conduct experiments on 20 class objects selected from COIL100 to evaluate the property of our method. Figure 4 shows how the average performance of SSA-AE varies with different λ. As we can see, when the parameter λ becomes larger, our method achieves better recognition at first and then the performance decreases. SSA-AE achieves the best performance when the λ is set to 5. It demonstrates that there exists a balance for the reconstruction error and the similarity constraint. Figure 5 shows how the average performance of SSA-AE varies with different stacked autoencoders number. From the results, we can observe that SSA-AE generally achieves better performance when the stacked autoencoders number goes up. However, we also notice that a much deeper structure would ruin the recognition performance. Therefore, in the experiments, we use a three-layer structure to generate the evaluation features.

To further study the performance of our approach, we compare SSA-AE with the following popular dimension reduction methods, including subspace learning: PCA [Turk and Pentland, 1991], NPE [He et al., 2005], KPCA [Schlkopf et al., 1998]; dictionary learning: SCC [Cai et al., 2011]; nonnegative matrix factorization: SDNMF [Qian et al., 2016]. In addition, three variants of the conventional autoencoder including Sparse-AE [Boureau et al., 2008], Denoise-AE [Vincent et al., 2010] and Graph-AE [Yu et al., 2013] are added. The experiments with different numbers of used objects (20, 40, 60, 80 and 100) are conducted to evaluate the scalability of our method. Each compared method is tuned with parameters to achieve their best performance. For all four autoencoder based methods, we use the same network architecture, and report the results on the feature of the last stacked layer.

Table 1 shows a large improvement on the recognition rates by our algorithm. From the results, we can observe that our method achieves consistently good performance when the number of used objects varies from 20 to 100. And autoencoder based methods are better than those traditional dimension reduction methods.

4.2 Results onMNIST

Furthermore, we evaluate the SSA-AE's performance on the MNIST dataset, a standard digit classification benchmark with a training set of 60,000 labeled images and a test set of 10,000 labeled images. An advantage of unsupervised learning algorithms is the ability to use them in semi-supervised scenarios where labeled data is limited. Here we use a randomly chosen subset of NL = 100, 600, 1000, and 3000 labeled images and NU = 60K unlabeled images from the training and validation set. We compare our method with many state-of-the-art semi-supervised learning methods consisting of MTC [Rifai et al., 2011a], PL-DAE[Lee, 2013], WTA-AE [Makhzani and Frey, 2015], SWWAE dropout [Zhao et al., 2015], M1+TSVM [Kingma et al., 2014], RFM [Patel et al., 2016], DRMM [Patel et al., 2016] and LadderNetwork [Rasmus et al., 2015].

Results are shown in Table 2 for our method with comparisons to related work. The experimental results of these compared methods are extracted from their paper. In this case, the unsupervised features are still trained on the whole dataset, but the SVM is trained only on the N labeled points where N varies from 100 to 3K. Also, we compare the performance of a supervised deep Convnet [Lecun et al., 1998] trained only on the N labeled training points. Despite the fact that we use no other regularization nor any hyperparameter search, the SSA-AE performs comparably to other approaches that are deep and employ various kinds of additional regularization. As we can see, SSA-AE achieves consistently good performance when N varies in a large range.

4.3 Implementation Details

We use Torch to implement our approach. For each single autoencoder, we stacked a combination of convolutional layer, batch normalization layer, ReLU layer and pooling layer to form the encoder. For all the convolutional layers, the number of channels is 50, and filter size is 5 × 5 with stride = 1 and padding = 0. Except for the experiments which evaluating the stacked autoencoder numbers, to preserve the convolutional feature maps, filter size is 3 × 3 with stride = 1 and padding = 1. For the decoder, we utilize the deconvolutional layer to obtain an upsampled feature map. The deconvolution operation is exactly the reverse of convolution. On the top of the hidden codes, we append an embedding layer whose dimension is 160. And we adopt stochastic gradient descent (SGD) for optimization.

5 Conclusions

In this paper, we proposed a novel autoencoder based approach called Stacked Similarity-Aware Autoencoders (SSA- AE) for unsupervised feature learning. To capture the geometric information in the data, we introduce a novel regularization technique. Specifically, we first obtain the pseudo class label of each sample by clustering the input features. Then for those samples sharing the same category label, their hidden codes of the autoencoder will be constrained to gather together in the high-dimensional space. This similarity constraint is implemented based on an extension of the recently proposed center loss. Through this combined cost function of the center loss and the autoencoder reconstruction error, the learned feature representations will be more discriminative and compact. As a general framework, different similarity constraint could be applied to the latent representations of the autoencoder as well. What's more, a stacked architecture is introduced to improve the representation capacity. The experimental results on several benchmark datasets indicated the effectiveness of our SSA-AE framework. Although our approach learns effective discriminative features, it has to pay high computational cost in clustering, which limits its applicability to large-scale problems. In the future, we plan to investigate more efficient way to obtain the pseudo class labels used for implementing the similarity constraint.

Acknowledgments

This work was supported by the National Basic Research Program of China(973 Program) under Grant 2013CB336500.