Supervised Representation Learning: Transfer Learning with Deep Autoencoders

1 Introduction

Transfer learning focuses on adapting knowledge from an auxiliary source domain to a target domain with little or without any label information to build a target prediction model of good generalization performance. In the past decade, a lot of attention has been paid on developing methods to transfer knowledge effectively across domains [Pan and Yang, 2010]. A crucial research issue in transfer learning is how to reduce difference between the source and target domains while preserving original data properties. Among different approaches to transfer learning, the feature-based transfer learning methods have proven to be superior for the scenarios where original raw data between domains are very different while the divergence between domains can be reduced.

A common objective of feature-based transfer learning methods is to learn a transformation to project instances from different domains to a common latent space where the difference of the projected instances between domains can be reduced [Blitzer et al., 2006; Dai et al., 2007a; Pan et al., 2008; 2011; Zhuang et al., 2014].

Recently, because of the power on learning high-level features, deep learning has been applied to transfer learning [Xavier and Bengio, 2011; Chen et al., 2012; Joey Tianyi Zhou and Yan, 2014]. Xavier and Bengio [2011] proposed to learn robust features with stacked denoising autoencoders (SDA) [Vincent et al., 2010] on the union of data of a number of domains. The learned new features are considered as high-level features, and used to represent both the source and target domain data. Finally, standard classifiers, e.g., support vector machines (SVMs), are trained on the source domain labeled data with the new representations, and make predictions on the target domain data with the new representations. Chen et al. [2012] extended the work of SDA, and proposed the marginalized SDA (mSDA) for transfer learning. mSDA addresses two limitations of SDA: highly computational cost and lack of scalability with high-dimensional features. More recently, Joey Tianyi Zhou and Yan [2014] proposed a deep learning approach to heterogeneous transfer learning based on an extension of mSDA, where instances in the source and target domains are represented by heterogeneous features. In their proposed method, the bridge between the source and target domains with heterogeneous features is built based on the corresponding information of instances between the source and target domains, which is assumed to be given in advance.

Though the goal of previous deep-learning-based methods for transfer learning is to learn a more powerful feature representation to reduce the difference between domains, most of them did not explicitly minimize the distance between domains when learning the representation. Therefore, the reduction in difference between domains is not guaranteed with the learned feature representation. Furthermore, most previous methods are unsupervised, and thus fail to encode discriminative information into the representation learning.

In this paper, we propose a supervised representation learning method for transfer learning based on deep autoencoders. Specifically, the proposed method, named Transfer Learning with Deep Autoconders (TLDA), is shown in Figure 1. In TLDA, there are two encoding and decoding layers, respectively, where the encoding and decoding weights are shared by both the source and target domains. The first encoding layer is referred to as the embedding layer, where the distributions of the source- and target- domain data are enforced to be similar by minimizing the KL divergence [Kullback, 1987] of the embedded instances between domains. The second encoding layer is referred to as the label encoding layer, where the source domain label information is encoded using a softmax regression model [Friedman and Rob, 2010], which can naturally handle multiple classes. Note that, in the second encoding layer, the encoding weights are also used for the final classification model. In summary, there are three key features in our proposed TLDA:

Figure 1: The framework of TLDA

1. The encoding and decoding weights are shared across different domains for knowledge transfer.
2. The distributions of two domains are enforced to be similar in the embedding space.
3. The label information is encoded.

2 Preliminary Knowledge

In this section, we first review some preliminary knowledge that is used in our proposed framework. Note that frequently used notations are listed in Table 1, and unless otherwise specified, all the vectors are column vectors.

2.1 Autoencoders

The basic framework of autoencoder [Bengio, 2009] is a feed forward neural network with an input layer, an output layer and one or more hidden layers between them. An autoencoder framework usually includes the encoding and decoding processes. Given an input x, autoencoder first encodes it to one or more hidden layers through several encoding processes, then decodes the hidden layers to obtain an output ˆx. Autoencoder tries to minimize the deviation of ˆx from the input x, and the process of autoencoder with one hidden layer can be summarized as:

where f is a nonlinear activation function (the sigmoid function is adopted in this paper), W1 ∈Rk×m and W ′1 ∈Rm×k are weight matrices, b1 ∈Rk×1 and b′1 ∈Rm×1 are bias vectors, and ξ ∈ Rk×1 is the output of the hidden layer. Given a set of inputs {xi}ni=1 , the reconstruction error can be computed by ∑ni=1 ‖ˆxi - xi‖2 . The goal of autoencoder is to learn the weight matrices W1 and W ′1 , and the bias vectors b1 and b′1 by minimizing the reconstruction error as follows,

2.2 Softmax Regression

The softmax regression model [Friedman and Rob, 2010] is a generalization of the logistic regression model for multiclass classification problems, where the class label y can take more than two values, i.e., y ∈ {1, 2, ..., c} (where c ≥ 2 is the number of class labels.). For a test instance x, we can estimate the probabilities of each class that x belongs to as follows,

where ∑cj=1 eθ⊤ j x is a normalized term, and θ1, · · · , θc are the model parameters.

Given the training set {xi, yi}ni=1 , yi ∈ {1, 2, ..., c}, the solution of softmax regression can be derived by minimizing the following optimization problem,

where 1{·} is an indicator function, whose value is 1 if the expression is true, otherwise 0. Once the model is trained, one can compute the probability of instance x belonging to a label j using Eq. (4), and assign its class label as

2.3 Kullback-Leibler Divergence

Kullback-Leibler (KL) divergence [Kullback, 1987], also known as the relative entropy, is a non-symmetric measure of the divergence between two probability distributions. Given two probability distributions P ∈ Rk×1 and Q ∈ Rk×1 , the KL divergence of Q from P is the information lost when Q is used to approximate P [Liddle et al., 2010], defined as DKL(P ||Q) = ∑ki=1 P (i) ln( P (i) Q(i) ). In this paper, we adopt the symmetrized version of KL-divergence, KL(P,Q) = DKL(P||Q) +DKL(Q||P), to measure the divergence for classification problems, Smaller value of KL divergence indicates more similar of two distributions. Thus, we use the KL divergence to measure the difference between two data domains when they are embedded to the same latent space.

3 Transfer Learning with Deep Autoencoders 3.1 ProblemFormalization

Given two domains Ds, and Dt, where Ds={x(s) i , y(s) i }|ns i=1 is the source domain labeled data with x(s) i ∈ Rm×1 , and y(s) i ∈ {1, ..., c}, while Dt = {x(t) i }|nt i=1 is the target domain with unlabeled data. Here, ns and nt are the numbers of instances inDs andDt, respectively.

As shown in Figure 1, there are three factors to be taken into consideration for representation learning. Therefore, the objective to be minimized in our proposed learning framework for transfer learning can be formalized as follows,

The first term of the objective is the reconstruction error for both source and target domain data, which can be defined as,

where

The first hidden layer is called the embedding layer with an output ξ ∈ Rk×1 of k nodes (k ≤ m), a weight matrix W1 ∈ Rk×m, and a bias vector b1 ∈ Rk×1 . The output of first layer is the input for the second hidden layer. The second hidden layer is called the label layer with an output z ∈ Rc×1 of c nodes (equals to the number of class label), a weight matrix W2 ∈ Rc×k , and a bias vector b2 ∈ Rc×1 . Here, the softmax Regression is used as the regularization item on source domain to incorporate label information. In addition, the output of the second layer is used as the prediction results for target domain. The third hidden layer ˆ ξ is the reconstruction of the embedding layer with the corresponding weight matrix and bias vector ˆ ξ ∈ Rk×1 and W ′2 ∈ Rk×c , b′2 ∈ Rk×1 . Finally, ˆx is the reconstruction of x with ˆx ∈ Rm×1 , W ′1 ∈ Rm×k , and b′1 ∈ Rm×1 .

The second term in the objective Eq. (7) is the KL divergence of embedded instances between the source and target domains, which can be written as

The goal of minimizing the KL divergence of the embedded instances between the source and target domains is to ensure the source and target data distributions to be similar in the embedding space.

The third term in the objective Eq. (7) is the loss function of softmax regression to incorporate the label information of the source domain into the embedding space. Specifically, this term can be formalized as follows,

where θ⊤ j (j ∈ {1, ..., c}) is the j-th row of W2.

Finally, the last term in the objective Eq. (7) is an regularization on model parameters, which is defined as follows,

The trade-off parameters α, β, and γ are positive constants to balance the effect of different terms to the overall objective.

3.2 Model Learning

The minimization problem of Eq. (7) with respect toW1, b1, W2, b2, W ′2 , b′2 , W ′1 , and b′1 is an unconstrained optimization problem. To solve this problem, we adopt the gradient descent methods. For succinctness, we first introduce some intermediate variables as follows.

The partial derivatives of the objective Eq. (7) w.r.t. W1, b1, W2, b2, W ′2 , b′2 , W ′1 , and b′1 can be computed as follows respectively,

where W2j is the j-th row of W2, and nsj is the number of instance with the label j in source domain. As the partial derivatives of the objective Eq.(7) w.r.t. b1, b2, b′2 , b′1 are very similar to those of W1, W2, W ′2 , W ′1 , respectively, we omit the details due to the limit of space. Based on the above partial derivatives, we develop an alternatively iterating algorithm to derive the solutions by using the following rules,

where η is the step length, which determines the speed of convergence. The details of the proposed algorithm is summarized in Algorithm 1. Note that the proposed optimization problem is not convex, and thus there is no guarantee on obtaining an optimal global solution. To achieve a better local optimal solution of the proposed gradient descent approach, we first run SAE on all source and target domain data, and then use the output of SAE to initialize the encoding and decoding weights.

Algorithm 1 Transfer Learning with Deep Autoencoders (TLDA)

Input: Given one source domain Ds = {x(s) i , y(s) i }|ns i=1 , and one target domain Dt = {x(t) i }|nt i=1 , trade-off parameters α, β, γ, the number of nodes in embedding layer and label layer, k and c.

Output: Results of label layer z and embedded layer ξ.

1. Initialize W1, W2, W ′2 , W ′1 and b1, b2, b′2 , b′1 by Stacked Autoencoders performed on both source and target domains;
2. Compute the partial derivatives of all variables according to Eqs. (14), (15) (16) and (17);
3. Iteratively update the variables using Eq. (18);
4. Continue Step2 and Step3 until the algorithm converges;
5. Computing the embedding layer ξ and label layer z using (9), and then construct target classifiers as described in Section 3.3.

3.3 Classifier Construction

After all the parameters are learned, we can construct classifiers for the target domain in two ways. The first way is directly to use the output of the second hidden layer. That is, for any instance x(t) in the target domain, the output of the label layer z(t) = f(W2ξ(t) + b2) can indicate the probabilities of x(t) which class it belongs to. We choose the maximum probability and the corresponding label as the prediction. The second way is to apply standard classification algorithms, e.g., logistic regression(LR) [Snyman, 2005; Friedman and Rob, 2010] to train a classifier for source domain in the embedding space. Then the classifier is applied to predict class labels for target domain data. These two methods are denoted as TLDA1 and TLDA2, respectively.

4 Experimental Evaluation

In this section, we conduct extensive experiments on three real-world image data sets to show the effectiveness of the proposed framework. Two of the three datasets are on binary classification, and the rest one is on multi-class classification.

4.1 Datasets and Preprocessing

ImageNet Data Set 1 contains five domains, i.e., D1 (ambulance+scooter), D2 (taxi+scooter), D3 (jeep+scooter), D4 (minivan+scooter) and D5 (passenger car+scooter). Data from different domains come from different categories, e.g., taxi from D2 and jeep from D3, therefore this dataset is proper for transfer learning study. To construct classification problems, we randomly choose two from the five domains, where one is considered as the source domain and the other is considered as the target domain. Therefore, we construct 20 (P 2 5 ) transfer learning classification problems. Statistics of this dataset is shown in Table 2.

1 http://www.image-net.org/download-features

Corel Data Set 2 [Zhuang et al., 2010] includs two different top categories, flower and traffic. Each top category further consists of four subcategories. We use flower as positive instances and traffic as negative ones. To construct the transfer learning classification problems, we randomly select one subcategory from flower and one from traffic as the source domain, and then choose another subcategory of flower and another one of traffic from the remaining subcategories to construct the target domain. In this way, we can construct 144 (P 2 4 · P 2 4 ) transfer learning classification problems.

Leaves Data Set [Mallah and Orwell, 2013] includes 100 plant species that are divided into 32 different genera, and each specie has 16 instances. We choose four genuses with more than four plant species to construct 4-class classification problems, and use 64 shape descriptor features to represent an instance. Each genus is regarded as a domain. Similar to the construction of ImageNet dataset, we can construct 12 (P 2 4 ) 4-class classification problems.

4.2 Baseline Methods

We compare our methods with the following baselines,

• The supervised learning algorithm Logistic Regression (LR) [Friedman and Rob, 2010] without transfer learning.
• Transfer component analysis(TCA) [Pan et al., 2011], which aims at learning a low-dimensional representation for transfer learning. Here we also use Logistic Regression as the basic classifier.
• Transfer learning based on stacked autoencoders, the marginalized Stacked Denoising Autoencoders (mSDA) method [Chen et al., 2012].

Implementation Details: After some preliminary experiments, we set α = 0.5, β = 0.5, γ = 0.00001 and k = 10 for the ImageNet and Corel datasets, while β = 0.05, k = 5 and γ = 0.0001 for the Leaves dataset. For mSDA, we use the authors' source code 3 and adopt the default parameters as reported in [Chen et al., 2012]. For TCA, the number of latent dimensions is carefully tuned, e.g., for the Corel dataset, the number is sampled from [10, 80] with interval 10, and its best results are reported.

2 http://archive.ics.uci.edu/ml/datasets/Corel+Image+Features.

3 http://www.cse.wustl.edu/ mchen/

4.3 Experimental Results

All the results of these three data sets are shown in Figure 2 and Table 3. Figure 2 shows the results over the 20 classification problems on the ImageNet dataset, in which x-axis represents the index of the problems, and y axis represents the corresponding accuracy. From the figure, we have the following observations,

• TLDA is significantly better than LR on every problem, which indicates the efficiency of our proposed transfer learning framework.
• TLDA performs better than TCA, which shows the superiority of applying deep autoencoders to learn a good representation for transfer learning. TLDA also outperforms mSDA, which indicates the effectiveness of incorporating label information from source domain.
• LR performs slightly worse than mSDA, even better than TCA. This may be because on constructed cross-domain classification problems, it is not easy to make knowledge transfer success. This observation again validates the effectiveness of our methods.

We also divide the constructed problems into two groups: a first group consists of problems on which the classification accuracy of LR is lower than 70%, and the rest problems are considered as a second group. The lower of classification accuracy of LR in some certain indicate the higher degree of the difficulty in knowledge transfer. The averaged accuracy of these two group as well as the averaged accuracy over all problems on the three three datasets are reported in Table 3. We can find that the proposed methods perform better than all the compared algorithms on the both groups of problems, except for that on the Leaves dataset, the performance of TLDA1 is comparable with that of TCA.

4.4 Parameter Sensitivity

In this section, we investigate the influence of the parameters α, β and k in the objective Eq.(7). In this experiment, when tuning one parameter, the values of the rest two are fixed. Specifically, α and β are sampled from {0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, 100}, and k is selected from {10, 20, ..., 80}. We select 10 out of the 144 problems on the Corel dataset for experiment, and report the results in Figure 3. From the figure, we can observe that the performance of TLDA1 is relatively stable to the selection of α and β, while it decreases dramatically when the value of k is large. Thus we set α = 0.5, β = 0.5 and k = 10 to achieve good and stable results for the ImageNet and Corel datasets.

5 RelatedWork

Poultney et al. [2006] proposed an unsupervised method with an energy-based model for learning sparse and overcomplete features. In their method, the decoder produces accurate reconstructions of the patches, while the encoder provides a fast prediction of the code without the need for any particular preprocessing of the inputs. Vincent and Manzagol [2008] proposed Denoising autoencoders to learn a more robust representation from an artificially corrupted input, and further proposed Stacked denoising autoencoders [Vincent et al., 2010] to learn useful representations through a deep network.

Transfer learning has attracted much attention in the past decade. To reduce the difference between domains, two categories of transfer learning approaches have been proposed. One is based on the instance level, which aims to learn weights for the source domain labeled data, such that the reweighted source domain instances look similar to the target domain data instances [Dai et al., 2007b; Gao et al., 2008;

Xing et al., 2007; Jiang and Zhai, 2007; Zhuang et al., 2010; Crammer et al., 2012]. The other is based on the feature representation level, which aims to learn a new feature representation for both the source and target domain data, such that with the new feature representation the difference between domains can be reduced [Blitzer et al., 2006; Dai et al., 2007a; Pan et al., 2008; Si et al., 2010; Pan et al., 2011; Xavier and Bengio, 2011; Chen et al., 2012; Zhuang et al., 2014].

Among most feature-based transfer learning methods, only a few methods aim to minimize the difference between domains explicitly in learning the new feature representation. For instance, maximum mean discrepancy embedding (MMDE) [Pan et al., 2008] and transfer component analysis (TCA) [Pan et al., 2011] try to minimize the distance in distributions between domains in a kernel Hilbert space, respectively. The transfer subspace learning framework proposed by [Si et al., 2010] tries to find a subspace, where the distributions of the source and target domain data are similar, through a minimization on the KL divergence of the projected instances between domains. However, they are either based on kernel methods or regularization frameworks, rather than exploring a deep architecture to learn feature representations for transfer learning. Different from previous works, in this paper, our proposed TLDA is a supervised representation learning method based on deep learning, which takes distance minimization between domains and label encoding of the source domain into consideration.

6 Conclusion

In this paper, we proposed a supervised representation learning framework for transfer learning with deep autoencoders. In this framework, the well known representation learning model autoencoder is considered, and we extend it to a deeper architecture. Indeed there are two layers for encoding, one is for embedding, where we impose the KL divergence constrains to draw the two distributions of source and target domains similar. The other is label layer, by which we can easily incorporate the label information from source domain. Finally, we conduct a series of experiments on three real-world image data sets, and all the results demonstrate the effectiveness of the proposed methods.

Acknowledgments.

This work is supported by the National Natural Science Foundation of China (No. 61473273, 61473274, 61175052, 61203297), National High-tech R&D Program of China (863 Program) (No.2014AA015105, 2013AA01A606). Sinno J. Pan is supported by the NTU Singapore Nanyang Assistant Professorship (NAP) grant M4081532.020.